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Chapter  1.  Introduction/Problem  statement: 

This  research  examines  the  roles  policy  analysts  and  policy  research  may  play  in 
improving  the  effectiveness  of  systems  of  care  serving  at-risk  children  and  their  families. 

This  work  presents  four  separate  analyses  which  use  a  variety  of  quantitative  and  qualitative 
analytic  methods,  and  examines  how  they  can  be  integrated  to  improve  service  design  and 
outcomes  evaluations  in  community-based  child  and  family  social  service  systems.  In 
presenting  results  from  these  analyses,  this  dissertation  seeks  to  explicitly  examine  the  roles 
rigorous  policy  research  can  play  in  such  contexts,  the  limitations  and  challenges  of  applying 
these  new  analytic  techniques  in  public  social  service  settings,  and  seeks  to  offer  suggestions 
for  areas  policy  researchers,  systems  managers,  political  leaders,  clients,  and  community 
members  might  focus  future  work  in  this  field. 

Community-based  service  models  are  gaining  increasing  attention  as  dissatisfaction 
grows  with  traditional  hierarchical  organizational  structures  used  in  many  public  social 
service  agencies  [Kamerman  and  Kahn,  1990;  Schorr,  1997].  These  reforms  involve  many 
separate  strategies,  including  assigning  staff  at  the  neighborhood  level,  devolving  decision¬ 
making  authority  to  local  actors,  allocating  resources  to  specific  communities,  and  turning 
over  operational  responsibihties  to  non-governmental,  community-based  organizations.  This 
shift  in  service  approach  places  a  variety  of  demands  on  these  organizations,  not  the  least  of 
which  is  to  develop  and  use  data  sources  and  methodologies  that  can  allow  decentralized 
operations  to  fairly  and  effectively  serve  vulnerable,  multi-challenged  children  and  families. 
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This  dissertation  examines  four  initiatives  undertaken  at  the  nation’s  largest  child 
welfare  service  district,  the  New  York  City  Administration  for  Children’s  Services  (ACS),  as 
central  components  in  that  agency’s  reform  activities  from  1996  to  2000.  This  reform 
strategy  emphasized  strengthening  the  operations  of  this  complex  organization,  while  shifting 
services  into  integrated,  community-based  networks  of  care.  Over  this  period,  this  agency 
significantly  expanded  staffing  levels,  increased  training  and  compensation  levels  for  line 
caseworkers,  supervisors  and  managers;  implemented  case  conferencing  and  instant  response 
teams  and  dramatically  lowered  caseloads  in  child  protective  service  operations  charged  to 
investigate  allegations  of  child  abuse  or  neglect;  and  implemented  a  dramatic  shift  to 
community-based  services  via  completion  of  a  Request  for  Proposal  process  covering  over 
$600  million  in  foster  care  and  preventive  services  per  year  [Scoppetta,  2001]. 

The  papers  presented  in  this  dissertation  represent  a  range  of  projects  conducted 
within  ACS  under  the  aegis  of  the  agency’s  newly  created  policy  research  unit,  the  Office  of 
Management  Development  and  Research.  As  such,  the  projects  underl5dng  the  research 
presented  here  reflect  the  roles  and  responsibilities  assigned  to  this  agency’s  policy  research 
team  during  the  reform  process,  and  represent  clear  and  explicit  examples  of  the  work  policy 
researchers  may  be  expected  to  contribute  in  public  social  service  organizations  undergoing 
reform. 

In  addition  to  examining  methodological  and  policy-level  implications  of  the  specific 
quantitative  and  qualitative  analytic  initiatives  used  to  support  the  implementation  and 
evaluation  of  community-based  service  programs,  this  dissertation  will  seek  to  frame  the 
analyses  presented  here  in  the  context  of  the  contribution(s)  policy  research  and  policy 
researchers  may  offer  in  such  settings.  These  projects  assigned  to  ACS’  policy  research  shop 
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as  part  of  the  community-based  reform  strategy  which  are  reported  here  include  work  to 
examine: 

1.  Methods  to  shift  resource  allocations  in  Community-Based  child  and  family  service 
systems  using  Community  Risk  Scales. 

2.  Use  of  qualitative  case  review  tools  to  assess  service  needs  for  children  and  families 
within  community-based  service  networks. 

3.  Integration  of  Community  Need  measures  into  event  history  models  of  Outcomes  for 
children  in  foster  care. 

4.  Integration  of  quality  of  care  and  program  Compliance/Timeliness  measures  into  event 
history  models  of  Outcomes  for  children  in  foster  care. 

Results  from  these  projects  will  be  synthesized  to  identify  common  strategies  that  can 
be  used  to  guide  evaluators,  managers,  researchers,  and  advocates  working  in  or  affected  by 
agencies  contemplating  shifting  to  neighborhood-based  services.  While  the  research 
presented  in  these  projects  emerges  from  a  single  large  child  welfare  service  district,  the 
methods  and  data  sources  described  here  should  have  application  in  a  wide  range  of  social, 
educational  and  health  service  settings.  If  effective,  these  methods  could  help  service  systems 
target  available  resources  to  communities  with  greatest  need  and  identify  services  and/or 
providers  offering  the  most  effective  outcomes  for  all  chents.  Further,  in  considering  the 
roles  played  by  ACS’  policy  research  team  in  conducting  these  and  other  analytic  tasks  in 
support  of  the  reform  process,  this  research  may  offer  broader  insight  into  the  strengths  and 
limitations  of  such  work  in  these  settings. 
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The  research  outlined  here  offers  significant  potential  contribution  to  existing 
methodologies  for  examining  vulnerabilities,  service  needs  and  evaluating  outcomes  in 
community-based  child  and  family  services  [Kamerman  and  Kahn,  1990;  Duquette,  Danziger, 
and  Seefeldt,  1997].  These  issues  are  important  given  the  continuing  calls  for  improved 
accountability,  performance  and  outcomes  measures  facing  many  large,  public  child  welfare 
agencies.  These  questions  gain  special  import  at  a  time  when  many  of  these  systems  are 
simultaneously  implementing  shifts  away  from  traditional,  centrally  managed  systems  of 
organization,  and  decentralizing  responsibility  for  their  vital  services  to  community-based 
organizations. 

The  first  two  analyses  described  above  provide  quantitative  and  qualitative  approaches 
that  can  be  used  in  a  variety  of  health,  education,  and  social  service  settings  to  better  target 
available  resources  to  match  needs  in  particular  communities.  It  should  be  noted  that,  if 
successful,  such  work  would  have  at  least  indirect  impact  on  the  overall  effectiveness  of  these 
services  since  it  would  support  allocation  of  resources  that  would  maximize  the  marginal 
benefit  of  available  funds  [Samuelson  and  Nordhaus,  1997;  Arrow,  1971;  Bebbington  and 
Davies,  undated  (a)]. 

The  final  two  analyses  described  here  could  provide  valuable  insight  into  the  data 
sources  and  methodologies  available  to  evaluate  outcomes  in  community-based  child  and 
family  programs.  Specifically,  these  analyses  seek  to  expand  existing  quantitative  outcomes 
models,  integrating  them  with  measures  of  community  need  and  service  quality  [Wulczyn, 
Orlebeke  and  Melamid,  2000].  These  analyses  seek  to  incorporate  broader  measures  of  child 
and  family  needs  into  the  existing  models,  and  provide  validation  of  qualitative  review  tools 
independent  of  the  consensus  methods  used  in  their  development.  This  work  has  the  goal  of 
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identifying  measurable  indicators  that  could  signal  positive  outcomes  for  vulnerable  children 
and  their  families.  Simultaneously,  this  analysis  might  identify  other  data  that  could  be 
incorporated  into  the  quantitative  outcomes  models  to  reduce  unexplained  variance  and 
improve  validity  of  these  measures.  By  directly  relating  variables  from  standard  case  records 
to  well  designed  measures  of  program  outcomes,  this  research  may  provide  the  initial  steps 
towards  the  development  of  improved  assessment  technologies  which  could  be  used  with 
individual  children  and  families  in  a  variety  of  community-based  service  settings. 

As  can  be  seen  in  Figure  1,  the  analyses  presented  in  this  work  rely  on  data  from  a 
variety  of  sources  which  are  only  partially  overlapping  or  linkable.  Particularly  notable  are 
the  sources  of  data  used  in  the  final  two  analyses — while  the  paper  “Integrating  Community 
Need  into  Outcomes  Evaluations”  uses  the  foster  care  outcomes  and  community  need  scale 
data  only,  the  final  paper  “Integrating  Process  and  Quality  Data  in  Outcomes  Evaluations” 
supplements  these  two  sources  with  data  on  foster  care  program  Quality  of  Care  and 
Compliance/Timeliness  measures.  Also  notable  in  this  work  is  the  inability  to  integrate  the 
data  on  child  and  family  needs  from  the  Matching  Needs  and  Services  analysis  with  the  other 
datasets  at  this  time. 
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Figure  1.  Overview  of  Analyses  and  Data  Sources  used  in  this  Dissertation. 
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The  range  of  work  presented  here  may  contribute  to  a  broader  understanding  of  how 
services  can  be  designed  and  integrated  to  better  meet  the  multiple  and  complex  needs  of  at- 
risk  children,  families,  and  communities  [Kamerman  and  Kahn,  1990;  Schorr,  1997].  To  be 
effective,  integrated,  community-based  service  systems  require  comprehensive, 
comprehensible,  integrated  information  systems  and  research  methods  which  can  support  the 
design  of  services  that  appropriately  reflect  child  and  family  service  needs,  that  fairly  and 
appropriately  allocate  resources  to  support  these  services,  and  that  evaluate  services  to  ensure 
they  effectively  meet  the  needs  of  vulnerable  children  and  families.  Thus,  while  the  work 


Assessing  Need  and  Evaluating  Outcomes  in  Community-Based  Services 


6 


presented  in  this  dissertation  does  not  explicitly  address  the  structure,  design  and 
implementation  of  integrated,  community-based  service  models,  it  does  seek  to  explore  a 
variety  of  analytic  issues  and  methodologies  which  may  be  vital  in  supporting  the 
development  and  ensuring  the  effectiveness  of  such  models. 
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Chapter  2.  Background  and  Literature  Review 

Child  and  family  services  encompass  three  principal  areas  of  focus  [Pecora  et  al., 

1992;  Bumam  and  Melamid,  1995]: 

•  Protection—  investigation  of  reports  of  child  abuse  or  neglect  and 
determination  of  services  and  interventions  necessary  to  ensure  safety  of 
children; 

•  Prevention-  services  to  prevent  occurrence  or  recurrence  of  child 
abuse/neglect,  to  reduce  incidence  of  foster  care  placements,  and  to  preserve 
and  strengthen  family  integrity;  and 

•  Placement—  out-of-home  services  through  foster  care,  adoption  and/or  legal 
guardianship  for  children  unable  to  remain  with  their  families. 

Child  and  family  service  systems  across  the  United  States  are  increasingly  focusing  on 
the  community  context  of  their  services  [Robertson  and  Wier,  1998].  This  shift  toward 
community-based  services  is  reflected  in  such  national  reform  models  as  the  Annie  E.  Casey 
Foundation’s  Family-to-Family  model  and  the  Edna  McConnell  Clark  Foundation’s 
Community  Partnerships  for  Child  Protection  [Annie  E.  Casey  Foundation,  1999;  Edna 
McConnell  Clark  Foundation,  undated].  It  is  also  reflected  in  major  community-based  service 
reforms  in  Los  Angeles  and  New  York  City,  the  nation’s  two  largest  child  welfare  service 
districts  [Scoppetta,  1996;  Bock,  2001].  While  these  reforms  employ  different  strategies  and 
emphasize  different  aspects  of  child  and  family  service  practice,  they  all  focus  on  enhancing 
the  important  roles  that  community  residents  and  community-based  service  providers  play  in 
addressing  the  needs  of  vulnerable  children  and  their  families. 
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Incidence  of  Child  Maltreatment:  The  need  for  additional  work  in  this  area  is 


substantial,  and  child  abuse  remains  one  of  the  most  pernicious  public  policy  issues  in 
America.  Data  for  1998  show  that  2.8  million  reports  of  possible  maltreatment  were  made  to 
child  welfare  agencies  in  that  year;  approximately  two-thirds  of  these  reports  were 
investigated,  and  903,000  children  were  estimated  to  have  been  victims  of  abuse  or  neglect. 
The  incidence  rate  of  children  being  substantiated  as  victims  of  abuse  or  neglect  dropped  to 
12.9  per  1,000  children  in  1998,  continuing  a  downward  trend  that  began  in  1993.  [Federal 
Green  Book,  2000,  p.  53] 

The  3'^^  National  Incidence  Study  of  Child  Abuse  and  Neglect  estimates  that 
approximately  42  out  of  every  1000  U.S.  children  are  endangered  or  harmed  by  abuse  or 
neglect  in  a  given  year  [Sedlak  and  Broadhurst,  1996,  p.  3-18;  National  Center  on  Child 
Abuse  and  Neglect,  1988].  Unfortunately,  only  39  percent  of  abused  and  28  percent  of 
neglected  children  came  to  the  attention  of  child  protective  services  authorities  in  this  review, 
even  though  a  large  majority  of  children  either  endangered  or  harmed  had  been  known  to 
other  investigative  agencies  or  professional  service  providers,  such  as  schools,  hospitals  or 
police  agencies  (Sedlak  and  Broadhurst,  1996,  p.7-32).  An  even  more  disturbing  reflection  of 
the  stress  on  child  welfare  systems  is  the  apparent  inability  to  respond  to  rising  abuse  and 
neglect  rates:  the  share  of  children  suffering  harm  who  came  to  the  attention  of  child 
protective  services  dropped  by  a  statistically  significant  44  percent  investigation  rate  from 
levels  reported  in  earlier  incidence  studies,  with  even  greater  fall-offs  among  children  who 
were  most  seriously  harmed  (Sedlak  and  Broadhurst,  1996,  p.  7-19). 

Foster  Care  Placement:  As  can  be  seen  in  Figure  2,  Foster  care  caseloads  in  the 
United  States  grew  42  percent  from  1990  to  1999;  568,000  American  children  lived  in  out-of- 
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home  foster  care  at  the  end  of  1999  [Federal  Green  Book,  2000].  This  pattern  represents  a 
growth  trend  that  has  extended  for  15  years  [Children’s  Defense  Fund,  1998;  Department  of 
Health  and  Human  Services,  1999].  At  any  point  in  time  approximately  7.7  out  of  1000  U.S 
children  are  in  foster  care  placement  [Federal  Green  Book,  2000]. 

Figure  2.  Foster  Care  Population  and  Placement  Rates  Per  1000  Children  0-18  in  the  United 
States  (Source:  Federal  Green  Book  of  Entitlement  Programs,  2000) 
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Increasingly,  analysts  are  focused  on  the  “dynamics”  of  foster  care  populations — how 
children  move  into,  through,  out  of,  and  back  into  foster  care  [Wulczyn,  Bruner,  and  Goerge, 
1999].  Such  analyses  allow  deeper  understanding  of  foster  care  systems,  and  provide  much 
more  useful  information  for  policy  makers,  managers  and  advocates  interested  in  these 
systems.  One  implication  of  these  studies  is  to  clarify  the  prevalence  of  foster  care 
placement — Pecora  et  al.  [1992]  emphasize  that  the  total  number  of  children  who  experience 
foster  care  placement  within  a  given  year  can  be  20  to  50  percent  higher  than  would  be 
indicated  by  point  in  time  estimates  [Pecora  et  al.,  1992,  p.  323-4;  Tatara,  1989]. 
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While  the  majority  of  families  come  to  the  attention  of  child  welfare  agencies  as  a 
result  of  abuse/neglect  reports,  numerous  studies  reveal  a  more  complex  portrait  of  their 
characteristics.  Research  has  consistently  indicated  that  30  to  40  percent  of  children  in  foster 
care  have  serious  and  persistent  health  problems;  that  30  to  40  percent  have  diagnosable 
mental  illnesses;  and  that  as  many  as  80  percent  are  developmentally  delayed  [Halfon  et  al., 
1994].  Other  researchers  have  noted  that  70  percent  of  children  in  foster  care  come  from 
families  where  one  or  more  members  are  substance  abusers  [Roberts,  1999].  Research  has 
also  generally  revealed  the  significant  educational  challenges  facing  children  in  foster  care 
[Melamid,  Kovach,  Barth  and  Rubbo,  1999;  Conger,  1999];  that  adults  with  experiences  in 
foster  care  are  disproportionately  likely  to  enter  the  criminal  justice  system  [Widom,  1991],  to 
become  homeless  [Koegel,  Melamid  and  Burnham,  1991],  and  to  use  public  assistance 
[Courtney,  1994;  Courtney,  1998;  Burnham  and  Melamid,  1996].  It  is  also  widely  held  that 
children  of  color  are  disproportionately  represented  in  these  systems,  experience  poorer 
outcomes  and  receive  lower  quality  of  care  than  Caucasian  children,  though  few  studies 
adequately  control  for  the  impacts  of  economic  and  social  well-being  on  this  relationship 
[Courtney,  Barth  and  Park,  1996]. 

Funding  Child  Welfare  Services:  Not  surprisingly,  the  growth  in  abuse/neglect  and 
foster  care  placements  has  generated  large  increases  in  funding  for  child  welfare  services. 
Total  annual  Federal,  state  and  local  expenditures  on  child  welfare  services,  including  child 
protection,  preventive  and  foster  care  services,  are  projected  to  reach  $16  billion  in  2001 
[Child  Welfare  League  of  America,  undated].  Federal  funding  for  Child  Welfare  represents 
approximately  44  percent  of  total  expenditures  in  this  area,  and  is  derived  from  several  major 
sources,  including  entitlement  funding  for  Title  IV-E  Foster  Care  and  Adoption  Subsidy,  as 
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well  as  dedicated  budget  allocations  for  Independent  Living  and  Training  funds,  Title  IV-B 
Family  Support  Services,  and  Title  XX  Social  Service  Block  Grant  streams  [Green  Book, 
2000].  As  can  be  seen  in  Table  1,  Federal  expenditures  on  child  welfare  services  have  grown 
by  large  amounts  over  the  past  decade,  and  are  projected  to  continue  substantial  growth  in  the 


future. 


Table  1.  Federal  Child  Welfare  Expenditures,  all  Sources,  1990-2005  [Source:  Federal 
Green  Book  of  Entitlement  Programs,  2000]. 


Total  Federal 
Child  Welfare 
Expenditures 
_ ^Millions) 

Annual 

Percent 

Change 

1990 

_1 

1.912 

1991 

$ 

2.328 

21.8% 

1992 

A 

2.796 

20.1% 

1993 

A 

3.171 

13.4% 

1994 

A. 

3.522 

11.1% 

1995 

A 

3.989 

13.3% 

1996 

A 

4.153 

4.1% 

1997 

A 

4.884 

17.6% 

1998 

A 

5.027 

2.9% 

1999 

A 

5.491 

9.2% 

2000festiniate) 

A 

6.081 

10.7% 

200Kestimate) 

A 

6.911 

13.6% 

2002restimatel 

A 

7.216 

4.4% 

2003festimatel 

A 

7.766 

7.6% 

20041estimate) 

A 

8.456 

8.9% 

2005^estimatel 

A 

9.213 

9.0% 

Fiscal  and  Management  Reform  in  Child  Welfare;  One  element  clearly  recognized 
as  driving  the  need  for  system  reform  is  how  the  system  is  currently  financed,  and  the 
incentives  this  creates.  In  particular,  the  work  of  Fred  Wulczyn  has  highlighted  the  need  to 
consider  and  implement  fiscal  reform  strategies,  at  the  Federal,  State  and  local  levels 
[Wulczyn,  2000].  The  most  widely  discussed  of  these  fiscal  reform  strategies  are  various 
forms  of  managed  care  and  contract  based,  privatized  services  designed  to  introduce  greater 
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flexibility  in  how  funding  can  be  used  while  incentivizing  improved  outcomes  for  children 
and  families  [Wulczyn  and  Orlebeke,  2000;  Kahn  and  Kamerman,  1999]. 

Welfare  Reform  and  Child  Welfare:  As  noted  above,  these  rises  in  child  welfare 
abuse/neglect  reports  and  foster  care  placement  rates  have  occurred  in  the  context  of  substantial 
drops  in  public  assistance  case  loads.  As  can  be  seen  in  Table  2,  Public  Assistance  services  were 
provided  to  6.3  million  Americans  in  1999,  less  than  half  the  level  in  1992— the  largest  welfare 
caseload  decline  in  history,  leaving  the  U.S.  with  it’s  lowest  percentage  of  the  population  on 
welfare  since  1965  [U.S.  Department  of  Health  and  Human  Services,  2001].  While  some 
research  indicates  that  risk  of  foster  care  placement  is  related  to  child-  and  family-risk 
characteristics,  and  that  low  income  and  public  assistance  utilization  are  not  so  related  [Thieman 
and  Dail,  1997]  data  from  the  National  Incidence  Study  of  Child  Abuse  and  Neglect  show 
that  children  in  families  earning  less  than  $15,000  per  year  had  maltreatment  rates  22  times 
greater  than  those  from  families  earning  $30,000  or  more  [Sedlak  and  Broadhurst,  1996;  Schorr, 
1997]. 

Table  2.  Public  Assistance  Caseloads  and  Utilization  Rates  1992-1999  [Source:  U.S.  Department 
of  Health  and  Human  Services,  2001] _ _ _ _ 


Fiscal  years 

Estimated  U.S. 
Population  (OOO's) 

AFDC/TANF 

Recipients 

Percent  of  U.S. 
Population 

June  1992 

254.462 

13.625.342 

5.4 

257.379 

14.142.710 

5.5 

259.935 

14.225.651 

5.5 

June  1995 

262.392 

13.660.192 

5.2 

June  1996 

264.827 

12.644.915 

4.8 

June  1997 

267.346 

10.823.002 

4.0 

June  1998 

269.845 

8.778.815 

3.3 

June  1999 

272.286 

7.187.753 

2.6 

December  1999 

274.076 

6.274.555 

2.3 

Moreover,  the  pattern  since  1996  remains  striking:  Public  Assistance  caseloads  have 


fallen  49  percent,  while  the  number  of  children  in  foster  care  rose  an  estimated  61,000 
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children  or  12  percent  [Green  Book,  2001].  While  some  jurisdictions  have  taken  active  steps 
to  redirect  welfare  reform  to  strengthen  child  and  family  services  [Bems  and  Drake,  1999], 
the  rise  in  foster  care  placements  in  recent  years  would  appear  to  validate  concerns  raised  by 
many  in  the  child  welfare  connmunity  when  welfare  reform  legislation  was  considered 
[Courtney,  1998;  Bumam  and  Melamid,  1995].  Ultimately,  the  challenges  of  welfare  reform 
on  child  and  family  service  systems  may  fall  disproportionately  on  big  cities  which  have 
historically  had  the  largest  public  assistance  caseloads,  and  which  may  be  subject  to  unfunded 
mandates,  unworkable  programs,  and  inadequate  resources  imposed  by  State  oversight 
authorities  [Kahn  and  Kamerman,  1998]. 

Family  Preservation  and  Support  Services:  The  service  strategy  that  has  received 
the  most  attention  in  recent  years  is  probably  family  preservation  or  foster  care  prevention 
services.  Researchers  and  policy  advocates  widely  expressed  rationale  for  preventive 
community-based  child  and  family  support  services  is  that  they  can  better  ensure  safety  of 
children  in  at-risk  families,  reduce  disruption  in  children’s  lives  associated  with  foster  care 
placement,  and  reduce  costs  to  these  systems  [Ensign,  1991].  These  benefits  may  pertain  both 
to  children  who  may  face  continuing  risk  of  abuse  when  they  return  to  their  families  after 
being  in  foster  care  (over  50  percent  of  children  entering  foster  care  return  to  their  families), 
and  also  reduce  the  one  in  four  incidence  of  children  being  replaced  in  foster  care  after 
returning  home  [Wulczyn,  Bruner  and  Goerge,  1999].  Finally,  preventive  community-based 
child  and  family  services  are  seen  as  being  different  in  kind  from  traditional  child  welfare 
practices,  and  better  able  to  meet  the  overlapping  challenges  facing  vulnerable  children  and 
their  families  [Berliner,  1993;  Forsythe,  1992]. 
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However,  a  large  body  of  research  has  revealed  significant  limitations  on  the 
interpretability  of  findings  from  many  of  the  evaluations  conducted  in  this  service  area,  and 
have  even  led  to  strenuous  calls  to  shift  away  from  this  service  approach  [Saffran,  1997; 
MacDonald,  1994].  For  example,  after  reviewing  a  large  number  of  available  studies  Frankel 
[1988]  found  “  the  lack  of  control  groups,  combined  with  inadequate  description  of  service 
activities,  make  it  impossible  to  generalize  reliably  across  programs”  (p.l45).  Similarly, 
Gelles  [1993]  found  “no  data  that  met  even  the  most  minimum  standards  of  scientific 
evidence  that  support  the  claims  [that]  family  preservation  ‘works’”  (p.559).  Other 
researchers  note  that  the  failure  to  demonstrate  effectiveness  in  these  programs  may  partially 
be  attributed  to  weak  methodology,  including  small  sample  sizes,  poor  control  group  designs, 
or  inappropriate  analytic  methodologies,  but  also  likely  stems  from  lack  of  integrity  in  many 
models,  including  deficient  outcome  measures  and  inadequately  defined  program  intervention 
strategies  [Rossi,  1992;  Kelly  and  Bl5lhe,  2000;  Epstien,  1997;  Rubin,  1997;  Henegan, 
Horwitz  and  Leventhal,  1996;  Schuerman,  Rossi,  and  Budde,  1999;  Bruner  and  Scott, 
Undated]. 

Other  authors  argue  that  despite  the  above  methodological  limitations,  some  evidence 
of  effectiveness  is  demonstrable  in  the  published  evaluations  of  community-based  child  and 
family  service  programs.  McCroskey  and  Meezan  [1998],  while  acknowledging  that  the 
research  literature  offers  few  clear  patterns  of  preventive  services  reducing  risk  of  child 
maltreatment  or  foster  care  placement,  do  find  that  these  programs  appear  to  improve  child 
development,  family  functioning  and  parenting  behaviors.  Fraser,  Nelson  and  Rivard  [1997] 
reviewed  28  evaluative  studies,  and  concluded  that  family  preservation  services  can  reduce 
violent  youth  behaviors  and  improve  mental  health  outcomes  of  participating  youth.  Other 
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proponents  have  suggested  integrating  family  preservation  approaches  with  existing  service 
programs,  including  child  protective  services  [Walton,  1997],  kinship  foster  care  [Berrick, 
Barth  and  Needell,  1994],  and  child  care  services  [McCroskey  and  Meezan,  1998]. 

One  result  of  the  tension  in  the  research  literature  on  the  effectiveness  of  family 
preservation  and  prevention  services  has  been  a  redefinition  of  the  objectives  of  these 
services,  generally  narrowing  their  service  goals  [Courtney,  1997].  For  example,  Malluccio 
and  Whitaker  [1997]  argue  that  the  family  preservation/child  protection  debate  has  been  too 
greatly  focused  on  “either/or”  thinking,  and  that  they  may  reflect  practices  that  should  well  be 
included  in  any  child  and  family  service  setting,  including  provision  of  a  complete  continuum 
of  care,  and  whole-family  assessments  (p.5).  Wells  and  Tracy  [1996]  similarly  suggest  that 
the  rationale  of  preventive  services  as  focused  on  preventing  foster  care  placement  be 
abandoned,  and  replaced  by  a  focus  on  child  development  and  prevention  of  maltreatment. 
Kelly  (2000)  emphasizes  the  apparent  importance  of  particular  service  elements,  such  as 
emergency  resources,  transportation,  treatment  engagement  as  long  as  14  months,  reliance  on 
highly  skilled  providers,  and  targeting  older  children  for  service. 

Systemic  Reform  in  Child  Welfare:  In  many  respects,  the  prevention/child 
protection  debate  has  subsided  in  the  child  welfare  field  in  recent  years,  with  many  in  the  field 
increasingly  focused  on  systemic  reform  and  balancing  needs  for  all  at-risk  children  and 
families  served  by  these  systems  [Lindsey,  1994;  Congressional  Quarterly  Researcher,  1998]. 
Kamerman  and  Kahn  [1990]  studied  25  child  welfare  service  districts  and  programs,  finding 
them  highly  focused  on  child  protective  services.  However,  these  authors  also  observed  that 
“in  none  of  the  locations  studied,  is  there  anything  resembling  a  full  delivery  system 
encompassing  enough  of  any  major  innovation  and  a  full  range  of  strategies;  the  best  is  still 
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very  circumscribed”  (p.l42).  These  authors  conclude  by  calling  for  integrated,  community- 
based  care  networks,  providing  a  comprehensive  service  continuum,  effective  case 
management,  and  supported  by  the  combined  resources  of  different  funding  streams  available 
to  support  such  work  (p.l52). 

These  and  other  authors  highlight  several  of  the  major  areas  of  challenge  facing 
systems  seeking  to  create  integrated,  community-centered  networks  of  care  for  vulnerable 
children  and  families.  For  example  Kamerman  and  Kahn  [1990]  note  the  strong  impact 
categorical  funding  streams  and  lack  of  basic  data  on  services  needs  have  in  restricting  service 
integration.  Duquette,  Danziger  and  Seefeldt  (1997)  seek  to  explicitly  build  on  this  work, 
providing  a  framework  for  analyzing  child  welfare  legislation  and  policies,  but  also  bringing 
central  focus  on  such  challenges  as  improving  casework  practice,  enhancing  service 
standards,  strengthening  assessment  skills  for  caseworkers,  collaborating  and  coordinating 
with  other  service  systems,  and  developing  data  and  information  systems  that  are  coherent, 
integrated  and  accessible.  In  a  similar  vein,  Waldfogel  [2000,  2000a,  1998]  has  focused  on 
the  need  to  improve  the  ability  of  child  protection  systems  to  assess  child  and  family  service 
needs  and  provide  “differential  response”  to  meet  the  needs  of  families  with  different 
challenges,  relying  on  community-based  service  partners  to  engage  and  strengthen  families. 

Issues  in  Child  Welfare  Evaluation  Methodology:  Underlying  virtually  all  the 
above  referenced  system  reform  efforts,  including  development  of  methodologically  sound 
managed  care  systems,  are  calls  for  creating  comprehensive  data  on  the  needs  of  children  in 
care,  the  performance  of  service  programs  and  the  outcomes  experienced  by  children  and 
families  touched  by  these  programs.  Community-based  service  strategies  require  new  sources 
of  information  to  ensure  that  responsibilities  devolved  to  the  community  level  are  met,  and 
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that  no  child  or  family  “slips  through  the  cracks”  in  the  system  [Schorr,  1997;  Waldfogel, 
2000a]. 

While  many  leading  researchers  have  contributed  effectively  to  the  literature  on 
program  evaluation  in  child  and  family  service  settings,  few  studies  had  greater  impact  on 
program  design  and  policy  development  in  child  welfare  than  Fanshel  and  Shinn  s 
longitudinal  studies  of  outcomes  from  the  Casey  Family  Program  [Fanshel  and  Shinn  1978; 
Pecora  et  al.,  1992].  Fanshel  and  Shinn’s  work  revealed  clear  relationships  between  several 
basic  aspects  of  care  and  child  outcomes,  including  the  roles  visitation,  child  age  and 
ethnicity,  family  needs,  casework  quality  and  the  dynamics  of  a  child’s  movements  in  care. 
This  work  directly  influenced  the  development  of  some  of  the  most  basic  practices  underlying 
much  of  foster  care  today,  including  permanency  planning,  visitation  policies,  and  serving 
children  in  least  restrictive  settings  [Green  Book,  2001;  Pecora  et  al.,  1992].  However,  even 
Fanshel  and  Shinn  felt  their  comprehensive,  multi-year  study  was  unable  to  capture  vitally 
important  elements  of  the  impact  of  foster  care  placement  on  the  children  and  families  living 
through  it  and  recognized  the  need  for  more  focus  on  evaluation  methods  [1978,  p.  479]. 

Several  authors  have  suggested  frameworks  to  structure  and  guide  the  development  of 
high  quality  evaluations  of  child  and  family  services  [Connell  et  al.,  1995;  Fulbright- 
Anderson,  Kubisch  and  Connell,  1998].  Courmey  [1993]  highlights  the  importance  of  foster 
care  evaluations  including  measures  of  program  structural  characteristics  (including  physical 
plant,  staff  ratios,  and  staff  qualifications/training),  program  process  characteristics  (including 
timeliness  of  assessment  and  treatment  planning,  and  quality  and  timeliness  of  all  casework 
contacts  and  visitation),  and  child  and  family  outcomes  (including  reunification,  reentry, 
adoption,  and  transfer/absence  from  care  events)  as  well  as  client  behavioral,  emotional,  and 
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attitudinal  shifts  and  client  satisfaction  measures.  However,  this  author  acknowledges  that 
such  comprehensive  reviews  are  particularly  challenging  given  the  lack  of  child  and  family 
need  assessment  tools  with  test-retest  validity,  the  complexity  and  cost  of  comprehensive 
evaluation,  and  the  diversity  and  lack  of  standardization  in  case  reporting  practices  [Courtney, 
1993,  p.28;  Magura  and  Moses,  1986]. 

Their  are  a  number  of  reasons  for  the  relative  absence  of  child  and  family  well  being 
measures  in  these  evaluations,  including  the  difficulty  of  defining  and  measuring  clinical  and 
service  needs,  reluctance  on  the  part  of  researchers  and  program  managers  to  use  tools  which 
might  negatively  label  children  and  families  and/or  set  unrealistic  expectations  for  systems  to 
“fix”  children  and  families  with  severe,  complex  and  the  long-standing  problems  [Altshuler 
and  Gleeson,  1999,  p.l28].  Development  of  client  need  assessment  tools  and  implementation 
of  comprehensive  program  evaluation  systems  in  child  welfare  requires  researchers  and 
program  managers  to  balance  organizational  culture  and  evaluation  methodologies  [Usher, 
1995].  Ultimately,  development  of  comprehensive  need  assessment  and  risk  management 
systems  may  well  represent  the  most  substantial  challenge  facing  child  welfare  systems  today 
[Gambrill  and  Shlonsky,  2001]. 

Integrating  “Qualitative”  and  “Quantitative”  Methodologies;  While  much  of  the 
data  required  for  evaluation,  needs  assessment  and  resource  allocation  in  these  settings  might 
be  derived  from  administrative  data  systems,  any  reform  effort  requires  a  full  range  of 
information  and  reporting  resources  including  well  designed  and  implemented  clinical 
assessments  [Wulczyn,  Orlebeke,  and  Melamid,  2000;  Kahn  and  Kamerman,  1999;  Connell  et 
al.,  1995;  Fulbright-Anderson,  Kubisch  and  Connell,  1998].  As  such,  effective  evaluation  of 
complex  child  welfare  service  systems  and  programs  should  be  expected  to  require  some 
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combination  of  objectively  observable,  quantitative  data,  and  more  subjective,  judgment- 
based  qualitative  data. 

Examples  of  quantitative,  objectively  observable  data  in  this  arena  include  child 
outcomes  from  care  (such  as  returning  home,  returning  to  care,  or  being  adopted  from  foster 
care),  child  and  family  demographic  and  service  history  characteristics,  or  administrative 
reports  of  compliance  with  and/or  timeliness  of  program  operations  [Fraser,  Pecora,  and 
Popuang,  1992;  Wulczyn,  Orlebeke,  and  Melamid,  2000].  Qualitative  or  subjective  data  on 
child  and  family  services  may  include  information  from  case  record  reviews,  clinical 
assessments,  site  observations  of  program  operations,  structured  interviews  with  clients  and/or 
staff,  and  survey  data  [Hodges,  1997;  Kirk  and  Ashcraft,  1998]. 

The  two  most  comprehensive  sources  of  data  available  in  most  child  welfare  service 
systems  are  administrative  data  and  case  records.  Since  administrative  data  systems  in  child 
welfare  agencies  typically  have  only  minimal  qualitative  data  on  the  service  needs  and  life 
experiences  of  the  children  and  families  served  by  these  agencies,  they  can  be  only  a  starting 
point  for  understanding  the  complex  and  dynamic  needs  and  experiences  of  the  children  and 
families  who  typically  come  to  the  attention  of  child  welfare  authorities  [Wulczyn,  Orlebeke, 
and  Melamid,  20(X)].  Supplementing  administrative  data  with  qualitative  information 
provides  a  “three  dimensional”  portrait  of  the  challenges  facing  families,  and  of  the  service 
strategies  that  might  better  meet  these  challenges  [Usher,  Gibbs  and  Wildfire,  1995;  Altshuler 
and  Gleeson,  1999]. 

Collecting  valid  qualitative  data  on  the  multi-dimensional  and  changing  issues  facing 
at-risk  children  and  their  families  is  difficult  and  costly,  requiring  careful  attention  to 
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methodology,  which  can  reduce  the  scope  of  what  can  be  reliably  examined.  Even  when  high 
quality  qualitative  information  is  collected,  results  can  be  difficult  to  summarize  and 
communicate  to  line  staff  and  clients.  Finally,  these  approaches  rarely  reflect  the  voice  and 
perspectives  of  line  staff,  managers,  parents,  children  and  others  who  have  substantial  day-to- 
day  experience  in  how  child  welfare  services  work,  and  the  needs  of  children  and  families 
served  by  these  systems. 

Themes  Emerging  from  the  Research  Literature 

The  research  reviewed  in  this  chapter  presents  a  number  of  significant  themes  which 
indirectly  or  directly  underlay  the  research  presented  in  this  dissertation.  Perhaps  the  broadest 
theme  emerging  from  the  literature  is  the  sizable  incidence  of  child  abuse  and  neglect  and 
foster  care  placement  in  the  United  States,  which  has  led  to  increased  public  attention  and 
funding  for  this  service  domain.  Research  also  consistently  demonstrates  that  vulnerability  is 
not  randomly  distributed  among  U.S.  children“families  and  children  living  in  poverty,  having 
health  and  mental  health  needs,  involved  in  substance  abuse,  homeless,  or  touched  by  the 
juvenile  justice  system  may  all  have  significantly  increased  risk  of  experiencing  abuse/neglect 
and  foster  care  placement. 

Also  notable  across  the  research  presented  here  is  significant  debate  on  how  best  to 
design  and  structure  services  to  at-risk  children  and  their  families.  While  there  are  many 
conflicting  voices  in  this  work,  there  does  appear  to  be  support  for  the  potential  for  integrated, 
community-based  services  to  provide  marginally  effective  services  to  families  impacted  by  a 
variety  of  service  needs. 
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The  research  literature  also  highlights  the  importance  of  rigorous  evaluation  methods. 
Courtney  [1993]  offers  a  comprehensive  review  of  the  elements  vital  to  effective  evaluation  in 
child  and  family  services,  including  consideration  of  program  structures  and  service  process 
in  evaluating  child  and  family  outcomes.  Courtney  also  highlights  the  importance  of 
adjusting  models  to  reflect  patterns  of  risk  and  need  for  all  children  and  families  served  by 
these  programs,  including  client  behavioral,  emotional  and  altitudinal  strengths  and 
challenges,  as  well  as  measures  of  client  satisfaction.  The  final  two  analyses  presented  in  this 
dissertation  explicitly  address  several  aspects  of  Courtney’s  evaluation  model,  including  the 
roles  risk  adjustment  may  assume  in  program  evaluation,  and  how  including  measures  of 
administrative  process  and  service  quality  may  affect  observed  foster  care  outcomes. 

However,  the  measures  used  in  the  evaluation  work  presented  here  fall  considerably  short  of 
the  kind  of  comprehensive  child  and  family  well-being  and  client  satisfaction  measures 
contemplated  by  Courtney  and  other  researchers. 

Given  the  observed  lack  of  exemplars  of  comprehensive,  integrated,  demonstrably 
effective  community-based  service  systems,  a  number  of  leading  researchers  in  the  field  have 
explicitly  sought  to  identify  areas  of  research  which  could  provide  more  context  and  direction 
on  the  range  of  issues  important  when  undertaking  reform  in  child  and  family  service  systems 
[Kamerman  and  Kahn,  1990;  Duquette,  Danziger  and  Seefeldt  (1997);  Waldfogel  (2000)]. 
These  researchers  point  to  a  range  of  issue  areas  which  must  be  considered  in  systems  reform, 
including  attention  to  funding  streams,  administrative  data  and  other  information  systems, 
casework  practices,  service  standards,  assessment  skills  and  techniques,  collaboration  and 
coordination  across  service  systems,  strengthening  child  protection  service  systems  and 
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designing  and  implementing  “differential  response”  structures  to  meet  child  and  family  needs 
comprehensively  and  effectively. 

The  analyses  presented  in  this  paper  address  certain  of  these  systems  design  and 
systems  reform  issues,  including  resource  allocations  and  design  of  service  structures  and 
programs  that  can  best  meet  child  and  family  needs.  However,  a  range  of  other  issues 
identified  in  the  research  literature  as  important  to  community  based  systems  reforms  are  not 
explicitly  addressed  in  the  work  presented  here.  The  most  notable  “gaps”  in  what  is  discussed 
in  this  work  relative  to  the  research  literature  seem  to  concentrate  around  the  domains  of 
implementing  and  managing  social  service  programs,  including  such  issues  as  clinical 
assessment  technologies,  case  management  practices,  and  staff  and  organizational 
development.  In  a  sense,  then,  the  range  of  projects  presented  here  may  represent  the  “front 
end”  of  the  reform  process,  including  resource  allocations  and  strategic  service  design,  and 
the  “back  end”  of  the  reform  process  as  reflected  in  program  evaluations,  but  have  less  to 
offer  regarding  the  actual  implementation  and  management  of  these  programs. 
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Chapter  3.  Research  Questions  and  Elements  of  this  Research 

Background:  The  research  presented  here  arises  from  four  analytic  approaches  used 
in  the  nation's  second  largest  child  welfare  service  district,  the  New  York  City  Administration 
for  Children's  Services  (ACS).  As  noted  above,  these  analyses  were  each  undertaken  under 
the  aegis  of  ACS’  Office  of  Management  Development  and  Research,  as  central  elements  in 
the  agency’s  community-based  service  reform  initiative,  from  1996  to  2000  [Scoppetta,  1996; 
Melamid,  Kovach,  Barth  and  Rubbo,  1998].  This  office  was  newly  created  under  this  reform 
initiative,  and  held  lead  agency  responsibility  for  policy  research,  data  analysis,  program 
evaluation  and  program  development  within  ACS.  At  its  most  basic  level,  then,  the  rationale 
underlying  why  these  specific  projects  are  presented  in  this  work  is  that  these  projects  are  the 
work  policy  researchers  were  “asked”  to  undertake  as  part  of  the  reform  process. 

In  fact,  the  projects  reflected  in  this  dissertation  reflect  a  subset  of  the  key  activities 
associated  with  the  reform  process  in  this  large  jurisdiction,  a  process  whose  complete  scope 
was  developed  through  intensive  consultation  with  a  wide  variety  of  system  stakeholders, 
including  political  leaders,  advocacy  and  research  organizations,  client  advocacy 
organizations,  and  plaintiffs  in  a  variety  of  legal  actions  relevant  to  the  agency.  The  selection 
of  projects  presented  in  this  dissertation  thus  represent  certain  key  responsibilities  assigned  to 
ACS  policy  researchers  in  this  reform  effort.  In  a  sense  then,  the  scope  of  work  presented 
here  directly  reflects  the  judgment  of  agency  leaders  in  this  jurisdiction  as  to  the  appropriate 
roles  policy  research  can  or  should  play  in  such  reform  efforts,  informed  by  the  values  a  wide 
range  of  actors  involved  in  the  reform  process  hold  towards  the  roles  policy  research  plays  in 
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such  settings.  These  analyses  are  summarized  in  the  following  four  chapters  of  this 
dissertation. 

Contrasted  against  the  themes  identified  in  the  literature,  the  scope  of  policy  research 
initiatives  undertaken  in  this  jurisdiction  seems  to  reflect  several  of  the  central  themes 
emerging  from  the  above  review  of  the  hterature  in  this  area,  including:  (a)  attention  to  the 
allocation  of  funding  and  staff  resources  to  support  community-based  networks  of  care;  (b) 
strategic  planning  to  design  services  appropriate  to  client  needs  and  risks,  and  (c) 
incorporating  risk  adjustment  and  service  process  measures  within  program  outcome 
evaluations  of  community-based  service  systems.  However,  the  literature  appears  to  suggest 
a  number  of  areas  of  research  focus  that  are  not  explicitly  reflected  in  the  projects  undertaken 
by  ACS  policy  research  staff.  In  particular,  the  projects  presented  in  this  dissertation  appear 
to  be  only  indirectly  relevant  to  many  issues  central  to  implementation  and  management  of 
service  programs  suggested  within  the  literature,  including  as  casework  practices,  staff 
training  and  development,  service  standards,  and  clinical  assessment  technologies. 

Conceptual  Framework;  Within  the  research  literature  cited  above,  the  most 
comprehensive  framework  for  developing  and  examining  evaluation  strategies  in  child 
welfare  settings  appears  to  be  the  work  of  Courtney  [1993].  Figure  2  aligns  the  multiple 
sources  of  information  included  in  this  dissertation  to  this  domains  identified  in  Courtney’s 
evaluation  framework.  As  can  be  seen,  the  data  to  be  presented  in  this  work  largely  align  with 
the  elements  of  Courtney’s  comprehensive  indicator-based  approach  to  measuring  outcomes 
in  Foster  Care  services,  though  many  of  these  measures  are  identifying  areas  that  are  and  are 
not  encompassed  within  the  Courtney  framework. 
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Figure  2.  Comparison  of  Research  Projects  presented  in  this  dissertation  with  Courtney’s 
Child  Welfare  Services  Framework  [Source:  Courtney,  1993]. 
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As  can  be  seen  in  the  last  column  of  Figure  2,  the  data  available  in  this  service  setting 
appears  to  align  well  with  several  of  the  domains  suggested  by  Courtney.  In  particular,  this 
agency  has  developed  a  series  of  event  history  survival  analysis  models  designed  to  provide 
comprehensive  Outcomes  Evaluations,  which  provide  a  range  of  Case  Status  Measures.  In 
addition,  these  models  are  able  to  control  for  many  Program  Structural  Characteristics,  as 
these  are  substantially  determined  by  the  service  program  in  which  children  are  placed,  as 
established  by  State  regulations  in  this  jurisdiction. 
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Research  Questions:  The  four  research  projects  presented  in  this  dissertation  each 
explicitly  seek  to  build  on  these  established  models,  exploring  methods  and  data  sources  by 
which  evaluators  may  more  comprehensively  assess  program  performance.  One  domain 
explored  in  this  work  is  the  incorporation  of  improved  “Client  Status  Measures”  into  child 
and  family  service  evaluations.  This  domain  is  explicitly  explored  in  each  of  the  projects 
described  here,  including  the  development  of  community-level  risk  scales,  and  review  of  a 
qualitative  case  review  tool  which  endeavors  to  identify  system  level  patterns  of  need  among 
service  populations.  The  third  analysis  presented  in  this  paper  extends  this  analysis,  exploring 
the  use  of  these  community-level  risk  data  within  program  evaluation  models. 

A  second  domain  suggested  by  Courtney  explored  in  this  dissertation  are  “Program 
Process  Characteristics”.  The  final  analysis  presented  in  this  dissertation  uses  data  from  two 
program  evaluation  systems  operating  in  this  setting,  exploring  how  agency-level 
performance  in  the  areas  of  Compliance/Timeliness  and  Quality  of  Care  are  related  to  and/or 
effect  outcomes  experienced  by  children  and  families  served  by  these  systems. 

The  “Client  Satisfaction”  domain  is  not  explicitly  considered  in  the  analyses  presented 
here — though  ACS  has  attempted  to  develop  reliable  measures  in  this  area,  no  consistent  or 
reliable  data  are  available  in  this  service  agency  at  this  time  [Wells,  1999].  However,  the 
Matching  Needs  and  Services  method  does  include  “client  validation”  processes,  whose 
potential  role  in  providing  external  validation  for  agency  service  planning. 
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Overview  of  the  Research  Projects 


1)  Developing  Community  Risk  Scales  to  estimate  need  for  Community-Based  Child 
and  Family  Services. 

One  challenge  in  implementing  community-based  services  is  to  understand  risks  and 
vulnerabilities  facing  children  and  families  across  communities  within  a  service  district.  Such 
information  is  vital  not  only  to  support  equitable  allocation  of  staff  and  funding  resources  to 
communities  being  served,  but  also  to  deepen  understanding  of  the  needs  and  challenges 
facing  different  neighborhoods.  While  evaluation  of  community-based  services  has  received 
increasing  focus  in  recent  years,  little  attention  has  been  paid  to  the  issue  of  community-level 
needs  assessment  and  how  the  allocation  of  resources  maximizes  the  marginal  benefit  of  each 
dollar  invested,  equalizes  access  to  services  for  all  needy  families,  and  promotes  efficacy  of 
program  at  the  systems  level  [Samuelson  and  Nordhaus,  1997;  Arrow,  1971;  Bebbington  and 
Davies,  undated  (a)].  This  work  explores  methodological  and  programmatic  issues  in 
measuring  and  enhancing  service  needs  in  conununity-based  services,  using  a  case  example 
from  ACS. 

This  research  presents  two  interlocking  strategies  for  considering  service  needs  and 
funding  equity — ^resource  allocation  formulas  and  model  budgets  for  funded  programs.  The 
results  of  this  analysis  demonstrate  that  relatively  simple  analytic  techniques  can  be  effective 
in  measuring  risks  across  communities  and  in  improving  funding  equity  across  multiple 
providers  operating  within  a  community-based  service  network.  A  wide  range  of  social  and 
health  service  systems  could  use  these  approaches  to  ensure  that  available  resources  are 
targeted  to  communities  and  clients  with  the  greatest  measured  need. 

2)  Matching  Needs  and  Services:  an  Outcome-Based  Needs  Assessment  Tool  for 
Community-Based  Service  Systems 
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This  research  presents  results  from  a  strategic  planning  process  used  to  analyze  service 
needs  at  the  community  level.  The  Matching  Needs  and  Services  (MNS)  method,  developed 
by  the  Dartington  Social  Research  Unit  in  England,  seeks  to  identify  patterns  of  need  among 
children  and  families  in  a  given  community,  using  statistically  valid  sample  designs  and 
rigorous  methodology  [Dartington,  1999].  This  process  uses  small  teams  of  staff  from  social 
service  systems  to  review  a  representative  sample  of  case  records  from  a  given  community. 
Teams  seek  to  identify  primary  service  needs  of  a  defined  target  population  and  formulate 
recommendations  for  best  meeting  these  identified  service  needs.  It  is  explicitly  assumed  that 
the  MNS  review  will  inform  the  structure  and  operation  of  the  services  delivered  to  clients, 
and  that  including  the  voice  and  perspective  of  staff  and  clients  directly  involved  in  these 
systems  will  yield  more  effective  analysis  and  more  useful  data  to  support  service 
enhancements. 

This  paper  presents  an  overview  of  the  MNS  methodology,  as  well  as  a  detailed  case 
example  from  the  implementation  of  this  approach  by  the  Administration  for  Children’s 
Services  (ACS)  in  New  York  City,  which  is  using  the  MNS  methodology  as  a  central 
component  of  its  newly  implemented  Neighborhood  Based  Services  strategy  [Scoppetta, 
1996].  This  paper  presents  results  from  ACS’  pilot  review  conducted  using  the  MNS 
methodology,  and  examines  the  issues  involved  in  implementing  strategies  emerging  from 
this  planning  process.  Finally,  this  work  offers  recommendations  for  how  this  approach  may 
be  used  in  a  social,  health  and  education  service  settings  where  there  is  need  to  clarify  service 
needs  and  better  target  resources  to  meet  the  needs  of  vulnerable  children  and  families. 

3)  Effects  of  Community-Level  Needs  on  Foster  Care  Outcomes 
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The  impact  of  community-level  needs  on  outcomes  for  children  in  foster  care  is 
explored  by  building  upon  evaluation  tools  previously  developed  by  the  New  York  City 
Administration  for  Children’s  Services  (ACS)  [Wulczyn,  Orlebeke  and  Melamid,  2000; 
Melamid,  20(X)].  These  measures  use  event  history  analysis  to  evaluate  the  amount  of  time 
children  spend  in  care  prior  to  experiencing  (a)  reunification  with  their  family,  (b)  adoption  or 
(c)  return  to  care  after  having  been  reunified.  This  analysis  augments  these  evaluation 
models,  adding  calculated  community  need  scores  derived  from  the  above  equity  analysis,  for 
each  of  59  community  districts  within  the  service  area  of  this  agency  for  each  child  reflected 
in  these  base  models. 

Incorporating  community  need  into  these  models  may  offer  insight  into  both  policy- 
level  and  methodological  issues  that  arise  in  the  implementation  and  evaluation  of 
community-based  foster  care  services.  Methodologically,  this  analysis  explores  the  use  of 
these  community  need  scores  as  a  proxy  for  child  and  family  level  risk  adjustment  to  the 
original  evaluation  model.  At  the  policy  level,  this  analysis  will  seek  to  assess  the  impact  of 
community  context  on  the  outcomes  children  and  families  experience  in  foster  care. 

4)  Integrating  Qualitative  and  Quantitative  Data  in  Evaluating  Social  Services 

As  noted  above,  there  is  wide  recognition  among  those  interested  in  child  and  family 

services  of  the  lack  of  consistent,  reliable,  meaningful  evaluation  in  many  programs  [Fraser, 

Pecora,  and  Popuang,  1992;  Bruner  and  Scott,  Undated;  Frankel,  1988].  Calls  for  greater 

accountability  are  rising  among  many  child  and  family  service  systems,  including  greater 

emphasis  on  “managed  care”  strategies  that  can  better  ensure  that  funds  expended  to  care  for 

vulnerable  children  are  used  in  the  most  effective  manner  possible  [Westat,  1995].  Such 

accountability  requires  accurate,  timely  and  complete  data  and  evaluation  methods  that  can 
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help  these  systems  better  understand  what  is  working,  for  whom,  and  at  what  cost  [Courtney, 
1993]. 

A  basic  tenet  of  social  science  program  evaluation  is  triangulation — the  use  of 
multiple,  well-designed  measures  to  enhance  and  confirm  the  validity  of  evaluation  scores  for 
any  subject  [Campbell  and  Fiske,  1959;  Crano,  1981;  Greene  and  McClintock,  1985; 
McClintock  and  Greene,  1985].  Combining  multiple  approaches  provides  more  accurate 
overall  evaluation  results,  and  provides  a  more  complete  portrait  of  how  services  work, 
making  results  more  useful  to  staff,  management,  clients  and  advocates.  While  data  available 
in  this  setting  will  not  support  a  comprehensive  analysis  of  the  power  of  triangulation  in 
evaluation  in  these  settings,  this  work  will  consider  the  issues  involved  in  evaluating  foster 
care  services,  to  identify  the  broad  strategies  available  to  managers,  evaluators,  advocates  and 
clients  who  wish  to  better  understand  the  effectiveness  of  different  programs  they  are 
operating.  It  does  so  by  examining  and  comparing  data  from  three  evaluation  approaches 
used  by  the  New  York  City  Administration  for  Children’s  Services  foster  care  system: 

1.  Compliance/Timeliness  measured  by  ACS’  Contractor  Overall  Performance  Evaluation 
System.  Data  for  these  indicators  are  collected  through  administrative  review  processes 
across  several  management  units  within  ACS,  and  fall  into  three  broad  categories — 
timeliness  of  completion  of  casework  activities,  success  or  failure  in  achieving  key 
service  objectives,  and  assessments  of  fiscal  management. 

2.  Quality  of  Care  measures  collected  through  the  Program  Evaluation  System  (PES). 
Developed  over  many  years  by  external  researchers,  the  PES  seeks  to  identify 
“meaningful  differences  in  agency  performance”  using  methodologically  rigorous  case 
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record  reviews,  structured  interviews  and  field  observations  [Wells,  1999].  For  each 
reviewed  program,  the  PES  collects  hundreds  of  indicators  across  three  conceptual 
domains  or  subscales:  Basic  Welfare;  Normal  Growth  and  Development;  and  Process. 
These  data  are  collected  by  dedicated  teams  of  evaluators,  who  review  a  statistically 
valid  sample  of  case  records  in  each  foster  care  agency. 

3.  Quantitative  outcome  indicators,  jointly  developed  by  ACS  and  the  Chapin  Hall  Center 
for  Children  at  the  University  of  Chicago,  rely  on  administrative  data  to  evaluate  key 
outcomes  experienced  by  children  in  foster  care  [Wulczyn,  Orlebeke,  and  Melamid, 
2000].  These  measures  use  event  history  analysis  to  evaluate  the  amount  of  time  children 
spend  in  care  prior  to  experiencing  (a)  reunification  with  their  family,  (b)  adoption  or  (c) 
return  to  care  after  having  been  reunified.  These  models  control  for  demographic  and 
placement  history  characteristics  recorded  in  the  ACS  database,  producing  performance 
scores  for  every  reviewed  agency  for  specific  cohorts  of  children  in  care,  and  incorporate 
controls  for  community  needs  developed  in  the  paper  described  above. 

Conclusions,  Cross-Cutting  Issues  and  Future  Steps 

The  final  chapter  of  this  dissertation  summarizes  issues  identified  across  these  papers, 
focusing  on  major  themes  emerging  from  these  disparate  works.  One  set  of  cross-cutting 
themes  explored  in  this  final  section  focuses  on  methodological  issues  that  emerge  from  the 
variety  of  analytic  tools  and  methods  used  in  these  analyses.  In  particular,  these  themes  seek 
to  synthesize  key  issues  which  arise  as  analysts  seek  to  integrate  multiple  sources  of  both 
quantitative  and  qualitative  data  to  support  the  complex  and  dynamic  needs  of  designing, 
implementing,  managing  and  evaluating  community-based  child  and  family  services. 
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A  second  set  of  cross  cutting  issues  addressed  in  the  final  section  of  this  dissertation 
focus  on  the  policy  level,  seeking  to  identify  major  themes  relating  to  policies,  organizational 
structures  and  service  practices  suggested  by  these  papers.  While  the  analyses  presented  in 
this  dissertation  can  only  represent  a  small  share  of  the  issues  and  methods  available  to 
support  the  development  of  integrated,  community-based  services,  these  analyses  may  offer 
some  insight  into  and  support  for  strategies  that  can  better  meet  the  needs  of  vulnerable 
children,  families  and  communities. 

Finally,  the  concluding  section  of  this  work  will  offer  suggestions  for  future  work  in 
this  area  which  might  deepen  our  understanding  of  strategies  that  can  support  effective 
services  to  families  and  children  facing  complex,  multi-dimensional  and  changing  needs. 

This  section  will  also  consider  briefly  the  range  of  policy  research  presented  in  this 
dissertation,  and  the  roles  policy  researchers  played  and  could  have  played  in  the  broader 
reform  initiative  taking  place  in  this  agency  at  this  time. 
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Chapter  4.  Using  Community  Risk  Scales  to  Allocate  Resources  for  Community-Based 
Child  and  Family  Services 

Introduction 

Public  policy  researchers  have  increasingly  focused  on  the  complex  challenges  of 
evaluating  outcomes  in  community-based  service  programs  [Rossi,  1992;  Kelly  and  Blythe, 
2000;  Epstien,  1997;  Rubin,  1997;  Henegan,  Horwitz  and  Leventhal,  1996;  Schuerman,  Rossi, 
and  Budde,  1999;  Bruner  and  Scott,  Undated].  These  efforts  have  increased  recognition  of 
the  importance  of  high  quality  evaluation  in  community  service  initiatives,  and  also  raised 
awareness  of  the  many  challenges  facing  evaluators  in  this  area. 

One  aspect  of  evaluating  community-based  services  that  has  received  less  attention  is 
resource  allocation;  evaluating  how  the  distribution  of  funding  and  staff  resources  maximizes 
the  marginal  benefit  of  each  dollar  invested  in  a  given  social  program,  increasing  the  fairness 
of  access  to  community-based  services  for  all  needy  famihes  in  a  given  service  jurisdiction 
[Samuelson  and  Nordhaus,  1997;  Arrow,  1971;  Bebbington  and  Davies,  undated].  In  this 
sense,  then,  this  paper  considers  equity  as  reflected  in  inputs  or  resource  distributions  that  best 
match  community  indicators  of  need,  as  well  as  the  fair  and  participatory  nature  of  decision 
making  processes  which  effectively  and  transparently  leads  to  the  implementation  of  resource 
distributions  that  best  match  observed  and  agreed  upon  standards  of  need  across  service 
districts.  This  paper  explores  methodological  issues  in  measuring  and  improving  fairness  in 
resomce  allocations  for  community-based  service  programs,  using  an  example  from  the 
nation’s  largest  child  welfare  preventive  service  program. 

Preventive  child  welfare  or  Family  Preservation  programs  serve  families  with  children 
at  imminent  risk  of  foster  care  placement,  and  promote  more  timely  and  more  stable  return 
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home  for  children  currently  in  foster  care  [Forsythe,  1992].  Preventive  services  have  seen 
substantial  growth  in  recent  years,  fueled  by  a  132  percent  rise  in  Federal  funding  for 
preventive  service  programs  from  1990  to  2000  [Federal  Green  Book  of  Entitlement 
Programs,  2001].  Annual  Federal  funding  for  these  services  now  totals  $587  billion,  and  a 
1995  review  identified  60  major  preventive  programs  operating  in  22  states  [Westat  et  al., 

1995] . 

The  most  commonly  offered  rationale  in  support  of  these  services  is  that  they  reduce 
disruption  in  children’s  lives  and  save  money  by  preventing  foster  care  placements  [Ensign, 
1991].  Foster  care  caseloads  in  the  United  States  grew  42  percent  from  1990  to  1999; 
568,000  American  children  currently  live  in  out-of-home  foster  care  [Green  Book,  2001; 
Children’s  Defense  Fund,  1998;  Department  of  Health  and  Human  Services,  1999].  It  is  also 
notable  that  more  than  half  of  children  entering  foster  care  are  eventually  returned  to  their 
families,  and  almost  one  in  four  discharged  children  return  to  care  [Wulczyn,  Bruner,  and 
Goerge,  1999].  Preventive  service  advocates  argue  that  by  working  with  the  entire  family 
they  can  better  ensure  safety  of  reunified  children  and  reduce  risk  of  replacement  in  foster 
care  [Wells  and  Tracy,  1996].  Finally,  preventive  services  are  seen  as  being  different  in  kind 
from  foster  care,  and  as  better  able  to  meet  the  overlapping  challenges  facing  vulnerable 
children  and  their  families  [Forsythe,  1992].  While  the  majority  of  children  enter  foster  care 
as  a  result  of  abuse/neglect  petitions,  numerous  studies  reveal  a  complex  portrait  of  the 
characteristics  of  these  children  and  families.  [Halfon  et  al.,  1994;  Melamid  et  al.,  1999; 
Conger,  1999;  Widom,  1991;  Koegel,  Melamid  and  Burnham,  1991;  Burnham  and  Melamid, 

1996] . 
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While  equity  has  been  a  major  area  of  focus  in  education  finance  and  other  fields, 
there  has  been  less  attention  to  resource  allocation  in  social  services  [Berne,  1988,  Hertert, 
Busch,  and  Odden,  1994;  Odden,  1998].  Undertaking  research  into  equity  in  child  and  family 
service  systems  appears  to  present  two  key  challenges:  wide  disparity  in  values  and  beliefs  as 
to  the  goals  and  objectives  of  these  systems  across  the  diversity  of  actors  who  impact  on  or 
are  impacted  on  by  these  services,  and  significant  challenges  in  collecting  data  and  evaluating 
risk,  need  and  strengths  across  all  communities  and  populations  served  by  these  programs. 

These  issues  are  explored  here  through  two  approaches  developed  by  the  New  York 
City  Administration  for  Children’s  Services  (ACS)  in  distributing  preventive  service  funds  to 
community-based  contract  service  providers.  This  paper  argues  that  readily  accessible 
methods  can  be  used  in  many  health  or  social  service  settings  to  allow  available  funds  to  be 
targeted  to  families  and  communities  most  in  need. 

Methodology 

The  ACS  Neighborhood-Based  Services  initiative  offered  a  unique  opportunity  to  re¬ 
examine  service  allocations  across  59  community  districts  in  New  York  City.  This  effort 
focused  on  two  elements;  (1)  a  resource  allocation  methodology  to  promote  input  equity  by 
distributing  available  preventive  service  funds  to  conununities  in  accordance  with  their 
service  needs  relative  to  other  conununities,  and  (2)  a  “model  budget”  funding  system  to 
enhance  unit  cost  equity  across  all  provider  agencies. 

Community  Resource  Allocations;  ACS’  existing  allocation  formula  for  preventive 

service  slots  used  only  two  indicators— abuse  and  neglect  reporting  rates  and  foster  care 

placement  rates  for  each  community  district,  with  foster  care  placements  double  weighted. 

While  this  formula  reflects  the  key  areas  of  focus  for  preventive  services — ^reducing  the 
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incidence  of  child  abuse  and  neglect  and  of  foster  care  placements — they  were  seen  as  having 
important  limitations.  One  concern  surrounded  under-reporting  of  child  abuse,  particularly  in 
communities  with  large  immigrant  populations,  communities  underserved  by  mandated  child 
abuse  reporters,  or  where  language  or  cultural  barriers  mitigate  against  some  community 
members  reporting  vulnerable  children  to  the  authorities.  In  addition,  ACS  wanted  to  use 
broad  measures  of  community  need  to  reflect  the  full  range  of  challenges  facing  vulnerable 
children  and  families.  Finally,  using  a  broader  range  of  indicators  was  seen  by  ACS  as  a  way 
to  reflect  the  important  contributions  of  community-based  health  and  social  service  providers 
in  directly  and  indirectly  reducing  foster  care  placements  by  supporting  the  needs  of  at-risk 
families. 

A  variety  of  health  and  social  indicators  are  available  at  the  community  district  level 
in  New  York  City.  The  Citizen’s  Committee  publishes  one  very  useful  compilation  of  child 
well-being  indicators  biennially  for  Children,  a  leading  child  and  family  services  advocacy 
group  [Citizen’s  Committee  for  Children,  1997].  The  Citizen’s  Committee  report  aggregates 
scores  across  seven  child  well-being  scales  into  community  risk  quintiles,  assigning  each  of 
New  York  City’s  59  Community  Districts  a  separate  rating  on  each  scale.  These  scores 
provide  relative  rankings  of  child  well-being  in  each  community  district  across  the  seven 
scales. 

These  scales  presented  two  clear  advantages.  First,  while  social  indicators  or 
population  estimates  such  as  these  are  difficult  to  measure,  these  data  were  all  produced  by 
Federal,  State  or  City  administrative  bodies  but  were  aggregated  into  carefully  conceived 
scales.  This  provided  substantial  confidence  in  both  the  underlying  reliability  and  consistency 
of  the  indictors  used  in  this  analysis,  as  well  as  in  the  face  and  content  validity  of  these  scales. 
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Figure  1  presents  a  listing  of  the  indicators,  scales  and  weights  used  in  this  analysis.  The 
second  advantage  of  using  these  scales  was  undoubtedly  political.  Relying  on  the  Citizen’s 
Committee  analysis  of  need  allowed  ACS  to  use  an  independent,  replicable  approach 
developed  by  leading  children’s  service  advocates,  reducing  some  of  the  likely  resistance  to 
reallocations  in  this  vital  and  often  contentious  service  area. 


Figure  1;  Indicators  and  Weights  used  in  ACS  Resource  Allocation  Model 


Weight  in 
Need  Score 

Domain 

Indicators 

25% 

Safety 

• 

• 

• 

Abuse  and  Neglect  Reports  Per  1,000  Children 

Indicated  Abuse/Neglect  Reports  Per  1,000  Children 

Foster  Care  Placements  Per  1,000  Children 

20% 

Poverty 

• 

• 

• 

Percent  of  Children  Receiving  Public  Assistance 

Unemployment  Rate 

Percent  of  Households  Below  $10,000  Income 

20% 

Health 

• 

• 

• 

Infant  Mortality  Per  1,000  Births 

Percent  of  Infants  at  Low  Birth  Weight 

Percent  Mothers  With  No  Prenatal  Care 

15% 

Youth  Risk 

• 

• 

• 

Percent  of  Births  to  Teens 

Percent  16-19  Year  Olds  Unemployed  and  Not  Graduated 

Youth  Arrest  Rate  (16-20  Year  Olds) 

10% 

Education 

• 

• 

Percentage  of  Students  Grades  2-10  Below  Grade  Level-Reading 
Percentage  of  Students  Grades  2-10  Below  Grade  Level-Math 

1S% 

Community 

Life 

• 

• 

Felony  Reports  Per  1 ,000  Residents 

Abandoned  Buildings  Scheduled  for  Demolition 

1S% 

Environment 

• 

• 

Lead  Paint  Violations  Per  1,000  Children  (1-4  years) 

Hazardous  Incidents  Handled  Per  1 ,000  Children 

While  these  well-being  scales  were  seen  as  a  methodologically  sound  and  politically 


expedient  starting  point  for  this  analysis,  ACS  senior  management  felt  some  elements  of  the 
original  scales  needed  to  be  modified  to  better  match  the  needs  of  this  analysis.  Of  primary 
concern  was  the  child  safety  scale,  which  in  the  Citizen’s  Committee  analysis  included  child 

abuse  and  neglect  reports  and  reported  violent  felonies  per  1 ,000  children  in  the  community. 
Allocating  Resources  based  on  Community  Needs  38 


ACS  managers  noted  that  this  measure  failed  to  include  foster  care  placements  and  child 
abuse  and  neglect  indications,  important  measures  of  service  demand.  At  the  same  time  the 
measure  of  felonies  in  the  community  was  seen  as  a  less  direct  proxy  for  children’s  safety.  As 
a  result,  the  safety  scale  included  abuse  and  neglect  reports,  indicated  abuse  and  neglect  cases 
and  foster  care  placements  per  1,000  children  in  each  community.’ 

The  final  step  in  developing  the  community  needs  indicators  was  to  establish  a 
weighting  formula  for  the  seven  child  well-being  scales.  Weights  were  developed  using  a 
consensus-based,  peer  review  process  among  members  of  ACS  senior  staff,  who  rank  ordered 
the  seven  scales  by  their  perceived  relation  to  need  for  preventive  services.  Child  Safety  was 
given  the  highest  rank  order,  followed  by  poverty  and  health  scales,  youth  risks,  education, 
community  life  and  environment.  The  highest  ranked  scale  (safety)  was  given  a  weight  ten 
times  greater  than  the  lowest  (environment).  The  remaining  scales  were  assigned  declining 
weights  based  on  their  rank  order,  as  presented  in  Figure  1 .  Final  allocations  were  based  on 
these  weighted  scores,  adjusted  to  reflect  the  number  of  children  under  18  years  of  age  in  each 
community  district.  However,  to  avoid  excessive  service  impacts  in  any  community,  ACS 
made  an  administrative  decision  to  limit  funding  cuts  in  any  Community  District  to  no  more 
than  25  percent  regardless  of  the  results  derived  from  this  allocation  formula. 

Model  Budget:  As  noted  above,  a  second  objective  in  ACS’  Preventive  Service 
funding  reallocation  was  to  increase  unit  cost  equity,  equalizing  funds  paid  to  preventive 
providers  for  each  “slot”  of  service  to  one  family  for  one  year.  Historically,  ACS  preventive 
service  contracts  were  negotiated  individually  with  providers.  As  a  result,  the  costs  of 
preventive  service  slots  varied  substantially,  from  $2800  to  almost  $8700  per  slot  per  year. 

'  Percentage  of  clean  streets  in  each  community,  was  also  excluded  from  the  environment  scale  of  the  CCC 
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To  remedy  these  concerns,  ACS  program  development  and  fiscal  staff  developed  standard 
model  budgets,  including  setting  direct  personnel  and  indirect  overhead  rates  constant  across 
all  programs.  The  ACS  model  budget  also  allows  reimbursement  levels  to  vary  for  programs 
of  different  sizes,  generally  providing  slightly  lower  average  reimbursement  per  slot  for  larger 
programs.  For  example,  general  preventive  programs  serving  an  average  of  45  families  would 
have  an  average  cost  of  $6,877  per  family,  while  the  programs  serving  165  families  would 
receive  average  funding  rates  that  are  10.6  percent  lower  at  $6,148. 

The  model  budget,  which  was  facilitated  by  a  62  percent  rise  in  overall  funding, 
substantially  equalized  resources  for  ACS  preventive  programs  by  funding  standard  salary 
levels,  staffing  patterns  and  support  service  resources  for  all  agencies.  By  setting  salary  levels 
commensurate  with  those  in  directly  managed  public  programs  and  in  contract  foster  care 
agencies,  ACS  also  hoped  to  improve  staff  recruitment  and  retention. 

Development  and  Review  of  Resource  Allocations:  Throughout  the  process  of 
developing  and  implementing  the  Community  Need  Scales  and  Model  Budgets,  ACS  Policy 
Researchers  and  Agency  Managers  undertook  a  number  of  specific  actions  designed  to  clarify 
the  scope  and  communicate  the  implications  of  revised  resource  allocations  likely  to  emerge 
from  this  work  to  all  organizations  and  parties  likely  to  be  interested  in  or  affected  by  these 
changes.  The  first  set  of  such  steps  occurred  during  the  development  of  these  measures  ^the 
design  of  these  distribution  mechanisms  was  initially  developed  by  teams  of  managers  and 
staff  from  a  variety  of  ACS  operating  units,  including  policy  research,  budget,  preventive 
service  program  development,  and  community  planning  and  development  teams.  The  initial 


scales  of  the  Citizen's  Committee  analysis,  at  the  recommendation  of  ACS  Senior  Managers. 
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values  for  the  scale  weights  used  in  the  Community  Need  measures  were  developed  from  the 
mutual  consensus  of  the  ACS  managers  responsible  for  these  operating  divisions. 

The  design  and  rationale  of  both  initiatives  were  repeatedly  presented  to  ACS  Senior 
Managers  to  ensure  that  all  relevant  internal  parties  had  the  opportunity  to  address  concerns 
with  these  strategies  before  “going  public”.  In  addition,  several  leading  researchers  with 
substantial  familiarity  with  the  preventive  service  system  in  this  jurisdiction  provided  review 
and  comment  on  these  measures.  These  reviews  resulted  in  small  modifications  to  some  of 
the  specific  measures  used  in  the  calculation  of  need  scales  as  described  above,  but  scale 
weights  were  left  unchanged  from  those  developed  by  the  management  team  described  in  the 
previous  paragraph. 

Communications  with  non-profit  programs,  advocacy  organizations  and  community 
groups  affected  by  these  reallocations  were  somewhat  constrained  by  the  fact  that  ACS  was 
simultaneously  involved  in  developing  and  issuing  a  major  Request  for  Proposals  (RFP) 
designed  to  implement  its  neighborhood-based  service  program.  Under  City  procurement 
regulations,  any  discussions  regarding  information  that  would  be  material  to  this  RFP  had  to 
take  place  in  public  settings  at  which  all  interested  parties  could  participate.  Given  the  broad 
impact  of  these  reallocations  and  this  RFP,  it  is  not  surprising  that  approximately  500 
representatives  of  community  organizations  attended  these  public  meetings.  What  was 
surprising  was  the  reaction  of  participants  to  the  resource  allocation  and  model  budget 
formulas  presented:  substantial  support  for  these  revised  methodologies  was  expressed  by  the 
large  majority  of  participants,  and  even  some  representatives  of  organizations  likely  to  receive 
reduced  funding  under  the  revised  formulas. 


Results 
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Table  1  presents  a  summary  of  neighborhood  allocations  for  ACS  preventive  services 
prior  to  and  after  the  adoption  of  the  resource  allocation  formula  across  each  of  New  York 


City’s  five  boroughs.  As  can  be  seen,  the  increase  in  overall  funding  for  this  service 
generated  rising  service  allocations  in  all  boroughs  of  New  York  City,  ranging  from  an  1 1 
percent  increase  in  Manhattan  to  an  85  percent  rise  in  the  Bronx.  However,  this  rise  in  overall 
funding  could  otherwise  mask  the  notable  shifts  in  how  these  funds  are  distributed-the 
Bronx,  Brooklyn,  and  Queens  got  larger  shares  of  total  funds,  while  Manhattan  saw  a 
substantial  drop  in  its  share. 


Table  1.  Changes  in  Resource  Allocations  across  the  Five  Boroughs  of  New  York  City 
(Dollars  in  Millions). _ _ _ _ _ _ 


Actual 

Percent 

Percent 

Population 

Expected 

Change-  Change- 

Change 

Change 

Need  Ratio 

Prior 

Change- 

New 

New 

New 

Due  to 

VS. 

Allocation 

Funding 

Formula  F  ormula 

Allocation 

Formula 

Expected 

New 

Old 

Bronx 

$11.30  23.5% 

$7.10 

$9.60 

85.1% 

$20.90 

26.7% 

$2.50 

13.6% 

1.36 

1.19 

Brooklyn 

$16.40  34.1% 

$10.30 

$13.40 

81.6% 

$29.80 

38.1% 

$3.10 

11.6% 

1.06 

0.95 

Manhattan 

$12.40  25.7% 

$7.80 

$1.40 

11.2% 

$13.80 

17.6% 

($6.40) 

-31.7% 

1.21 

1.77 

Queens 

$6.70  14.0% 

$4.20 

$5.00 

74.1% 

$11.70 

15.0% 

$0.80 

7.3% 

0.62 

0.58 

Staten 

$1.30  2.7% 

$0.80 

$0.80 

64.0% 

$2.10 

2.7% 

$0.00 

0.0% 

0.48 

0.48 

Island 

$48.10 

$30.20 

$30.20 

$78.30 

Table  1  further  separates  these  changes  into  two  factors:  the  impact  of  the  overall 

$30.2  million  increase  in  funding,  and  the  application  of  ACS’  revised  allocation  formula. 

This  analysis  reveals  a  more  complex  image  of  the  impacts  of  this  new  methodology.  As  can 

be  seen,  the  relatively  small  net  increase  in  preventive  service  funding  in  Manhattan  results 

from  the  new  allocation  formula;  had  the  revised  formula  been  used  but  no  additional  funding 

been  available,  Manhattan  could  have  experienced  a  31.7  percent  drop  in  funding.  At  the 

other  extreme,  Bronx  and  Brooklyn  experienced  increases  in  total  funding,  driven  by  the 

changes  in  the  allocation  formula  compounding  the  large  increase  in  overall  funding.  On  net. 
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then,  one  can  see  that  the  ACS  allocation  formula  had  the  effect  of  shifting  resources  from 
Manhattan  to  the  outer-boroughs  of  New  York  City,  but  that  the  availability  of  substantially 
increased  funding  for  preventive  services  allowed  for  service  expansion  in  every  borough. 

Table  1  also  presents  population  need  ratios,  calculated  by  dividing  the  share  of 
preventive  service  funds  for  each  borough  by  the  share  of  the  total  City  population  under  18 
in  that  Borough.  This  ratio  provides  a  measure  of  the  intensity  of  need  for  preventive  services 
in  each  Borough.  Children  in  the  Bronx  have  on  average  a  need  for  services  that  is  36  percent 
greater  than  the  City  as  a  whole,  while  children  in  Manhattan  and  Brooklyn  have  needs  that 
are  21  and  6  percent  greater,  respectively.  At  the  same  time,  children  in  Queens  and  Staten 
Island  have  lower  average  need  levels  (38  and  52  percent  below  citywide  average  need, 
respectively). 

Table  2  presents  final  slot  allocations  across  the  five  boroughs,  estimating  the  impacts 
of  both  the  resource  allocation  formula  and  model  budgets."  Under  the  new  model  budget, 
average  reimbursement  to  preventive  service  providers  per  family  per  year,  grew  16.3  percent. 
Given  New  York  City’s  62.6  percent  funding  increase  for  preventive  services,  this  increase  in 
model  budget  costs  supported  an  estimated  39.8  percent  rise  in  the  number  of  families  served 
at  any  point  in  time.  This  allowed  for  the  creation  of  3,099  additional  service  slots  Citywide. 

On  balance,  the  combination  of  substantially  higher  overall  levels  of  funding, 
reallocation  of  these  funds  across  neighborhoods,  and  the  use  of  a  model  budget  formula  to 
equalize  resources  across  service  agencies  appear  to  generate  very  substantial  changes  in  the 
ACS  preventive  service  program.  The  Four  “outer  Boroughs”  (Bronx,  Brooklyn,  Queens  and 

"  Note  that  figures  reported  here  differ  slightly  from  ACS’  actual  fund  allocations,  which  are  affected  by  the 
actual  sizes  of  funded  agencies  and  are  subject  to  contract  negotiations.  In  addition,  borough  level  funding  for 
prior  allocations  were  not  available,  and  are  estimated  using  citywide  average  cost  per  slot. 
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Staten  Island)  experience  rises  in  service  slots  of  between  40  and  60  jiercent.  However, 
Manhattan  sees  a  drop  of  approximately  91  service  slots,  representing  a  decline  of  4.5  percent 
in  available  service  capacity  for  that  Borough. 

Table  2.  ACS  Service  Slots  by  Borough,  Estimated  Allocation  under  Prior  and  Revised 
Allocation  Formulas.  _ 


New 

Preventive 
Service  Slot 
Allocation 

Oid 

Preventive 
Service  Slot 
Allocation 

Change 

in  Service  Siots 
Aiiocated 

Bronx 

2,905 

1,829 

1,076 

Brooklyn 

4,145 

2,660 

1,485 

55.8% 

Manhattan 

1,916 

2,007 

(91) 

-4.5% 

Queens 

1,636 

1,092 

544 

49.8% 

Staten  Island 

293 

209 

84 

40.2% 

Citywide  Total: 

10,895 

7,796 

3,099 

39.8% 

Budget  per  Service  Slot: 

$  7,189 

$  6,179 

$1,010 

16.3% 

Finally,  Table  3  presents  data  on  the  allocation  of  service  slots  to  each  of  New  York 
City’s  59  community  districts,  focusing  on  how  slot  allocations  would  have  differed  had  the 
prior  allocation  formula  been  used  to  make  slot  awards  in  each  neighborhood.  This  analysis 
reflects  the  difference  between  the  slots  awarded  using  the  full  revised  resource  allocation 
formula  and  what  awards  would  have  been  had  an  allocation  formula  similar  to  the  prior 
methodology  been  used.  This  analysis  uses  the  child  safety  scale  within  the  new  allocation 
formula  as  a  proxy  for  this  older  model,  comparing  actual  slot  allocations  to  what  they  would 
have  been  had  a  safety-scale  only  model  been  used. 


Allocating  Resources  based  on  Community  Needs 


44 


Table  3.  Impact  of  New  Preventive  Service  Allocation  Models  across  New  York  City 


Community  Districts 


Name 

CD 

Slots 

Awarded 

Change 

from 

Old 

Model 

Percent 

Change 

Name 

CD 

Slots 

Awarded 

Change 

from 

Old 

Model 

Percent 

Change 

Bronx 

2905 

303 

11.6% 

Manhattan 

1916 

-88 

-4.4% 

Mott  Haven 

X 

1 

316 

34 

12.1% 

Battery  Park 

M 

1 

13 

-9 

-40.9% 

Hunts  Point 

X 

2 

168 

16 

10.5% 

G'wich  Village 

M 

2 

29 

-37 

-56.1% 

Morrisania 

X 

3 

232 

23 

11.0% 

Lower  E.  Side 

M 

3 

273 

21 

8.3% 

Concourse 

X 

4 

428 

19 

4.6% 

Chelsea 

M 

4 

51 

-38 

-42.7% 

Univ.  Heights 

X 

5 

436 

104 

31.3% 

Midtown 

M 

5 

14 

-6 

-30.0% 

E.  Tremont 

X 

6 

230 

55 

31.4% 

Murray  Hill 

M 

6 

24 

-17 

-41.5% 

Fordham 

X 

7 

276 

-12 

-4.2% 

Upper  W.  Side 

M 

7 

158 

-44 

-21.8% 

Riverdale 

X 

8 

70 

4 

6.1% 

Upper  E.  Side 

M 

8 

25 

-19 

-43.2% 

Soundview 

X 

9 

370 

98 

36.0% 

Manhattanville 

M 

9 

263 

-32 

-10.8% 

Throqs  Neck 

X 

10 

78 

-33 

-29.7% 

Central  Harlem 

M 

10 

332 

38 

12.9% 

Pelham  Pkwy. 

X 

11 

100 

-15 

-13.0% 

E.  Harlem 

M 

11 

407 

33 

8.8% 

Williamsbridge 

X 

12 

201 

8 

4.1% 

Washington  Hts. 

M 

12 

327 

21 

6.9% 

Brooklyn 

4145~^ 

343 

9.0% 

Queens 

1^6 

-271 

-14.2% 

Williamsburg 

K 

1 

270 

94 

53.4% 

Astoria 

Q 

1 

151 

-64 

-29.8% 

Fort  Greene 

K 

2 

167 

-25 

-13.0% 

Sunnyside 

Q 

2 

44 

13 

41.9% 

Bedford  Stuy. 

K 

3 

482 

62 

14.8% 

Jackson  Hts. 

Q 

3 

94 

-11 

-10.5% 

Busvvick 

K 

4 

342 

54 

18.8% 

Elmhurst 

Q 

4 

86 

33 

62.3% 

E.  New  York 

K 

5 

535 

102 

23.6% 

Ridgewood 

Q 

5 

104 

-18 

-14.8% 

Park  Slope 

K 

6 

267 

-56 

-17.3% 

Forest  Hills 

Q 

6 

33 

-2 

-5.7% 

Sunset  Park 

K 

7 

209 

22 

11.8% 

Flushing 

Q 

7 

67 

-29 

-30.2% 

N.  Crown  Hts. 

K 

8 

2^ 

-15 

-5.3% 

Fresh  Meadows 

Q 

8 

67 

13 

24.1% 

S.  Crown  Hts. 

K 

9 

211 

25 

13.4% 

Woodhaven 

Q 

9 

103 

-48 

-31.8% 

Bay  Ridge 

K 

10 

40 

-42 

-51.2% 

Howard  Beach 

Q 

10 

121 

-33 

-21.4% 

Bensonhurst 

K 

11 

72 

12 

20.0% 

Bayside 

Q 

11 

32 

-11 

-25.6% 

Borough  Park 

K 

12 

114 

25 

28.1% 

Jamaica 

Q 

12 

409 

-28 

-6.4% 

Coney  Island 

K 

13 

185 

-2 

-1.1% 

Queens  Village 

Q 

'13 

124 

-55 

-30.7% 

Flatbush 

K 

14 

201 

12 

6.3% 

Rockaways 

Q 

'14 

201 

-28 

-12.2% 

Sheepshead 

K 

15 

82 

24 

41.4% 

Brownsville 

K 

16 

355 

39 

12.3% 

Staten  Island 

293 

-212 

-42.0% 

E.  Ratbush 

K 

17 

222 

36 

19.4% 

Willowbrook 

;  S 

1 

196 

-110 

-35.9% 

Canarsie 

K 

18 

124 

-24 

-16.2% 

South  Beach 

S 

2 

57 

-69 

-54.8% 

Tottenville 

1  S 

;  3 

40 

-33 

-45.2% 

The  impacts  of  the  newer,  fuller  model  were  quite  substantial  in  several  communities. 


Three  community  districts  saw  a  change  of  more  than  100  service  slots  as  a  result  of  the  use 


of  the  full  model,  and  one-third  of  communities  experienced  a  switch  of  30  percent  or  more  in 


their  allocated  slots.  The  direction  of  change  due  to  these  models  also  offered  some 
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interesting  patterns.  Two  boroughs,  the  Bronx  and  Brooklyn,  gained  in  slot  allocations  under 
this  full  model,  while  the  other  three  boroughs  saw  declines.  In  addition,  there  are  some 
patterns  within  the  boroughs  that  are  worthy  of  note.  For  example,  while  Manhattan  did 
receive  4.4  percent  fewer  slots  under  the  full  model,  the  four  community  districts  with  the 
greatest  need  in  this  borough  saw  increases  (Lower  East  Side,  Central  Harlem,  East  Harlem 
and  Washington  Heights).  Other  needy  communities  receiving  increases  under  the  full  model 
include  much  of  the  South  Bronx  (community  districts  1  through  6  and  9)  and  Central 
Brooklyn  (community  districts  3, 4, 5, 9, 14, 16  and  17). 

Discussion 

The  revised  approach  to  allocating  preventive  services  in  New  York  City  appears  to 
have  had  substantial  impacts  on  the  system,  on  individual  providers  and,  hopefully,  on  the 
children  and  families  served  by  these  programs.  The  combination  of  putting  in  place  model 
budgets  and  revising  how  resources  are  distributed  among  communities  in  New  York  City 
should  equalize  access  to  quality  service,  relative  to  what  distributions  would  have  been  under 
previous  allocation  and  budgeting  methods  used  in  this  jurisdiction. 

However,  this  analysis  has  significant  limitations  which  must  be  acknowledged.  One 

set  of  limitations  surround  the  lack  of  independent  validation  for  these  measures,  other  than 

the  review  and  consensus  methodologies  used  to  confirm  indicators  used  and  weights 

assigned  to  scales.  Greater  validity  could  be  established  by  assessing  correlations  and 

predictive  value  of  these  and  a  variety  of  other  measures  against  relevant,  desired  outcomes 

for  children  and  families  served  by  these  systems,  such  as  extent  and  nature  of  violence  or 

neglect  in  families,  incidence  of  substance  abuse  or  emotional  disturbance  or  inadequate 

housing.  However,  many  of  these  desired  outcomes  or  needs  are  extremely  difficult  to 
Allocating  Resources  based  on  Community  Needs  46 


measure,  which  would  introduce  some  degree  of  error  into  any  external  validation 
calculations. 

Much  of  the  challenge  in  measuring  outcomes  in  child  and  family  services  also  arises 
from  the  different  beliefs  held  by  actors  in  these  issue  areas  as  to  what  constitutes  positive 
outcomes  for  these  services.  One  reflection  of  these  differences  was  the  substantial  debate 
among  research,  policy  and  advocacy  actors  as  to  the  role  preventive  or  family-preservation 
services  should  play  in  supporting  vulnerable  children  and  their  families — ^from  preventing 
foster  care  placement,  to  preventing  child  abuse  and  neglect,  to  improving  the  broadly  defined 
“well-being”  of  families  and  communities.  Each  of  these  sets  of  objectives  imply  different 
sets  of  outcome  measures  for  children  and  families  affected  by  these  programs,  ranging  from 
incidence  of  foster  care  placement  or  child  abuse/neglect,  to  difficult  to  define  child  and 
family  health  and  well-being  measures.  This  diversity  in  desired  outcomes  in  child  and 
family  services  introduces  significant  challenges  to  any  effort  to  conduct  more  formal  analysis 
of  equity — ^unless  explicit  consensus  can  be  reached  among  those  most  affected  by  these 
programs,  any  standard  used  to  analyze  equity  in  these  settings  will  be  open  to  challenge  and 
further  debate. 

The  consensus-based  decision  making  processes  used  in  undertaking  this  work  sought 
to  mitigate  some  of  these  limitations.  One  exemplar  of  the  role  consensus  processes  played 
here  was  the  decision  to  use  reports  developed  by  a  leading  advocacy  organization  in 
selecting  community  need  indicators.  Another  example  was  the  use  of  consensus  decision 
making  processes  in  determining  scale  weights  for  resource  allocations.  Finally,  model 
budgets  sought  to  establish  parity  across  organizations  in  funding  levels.  Each  of  these 
approaches  appears  methodologically  sound,  and  offered  the  additional  benefit  of  increasing 
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buy-in  from  other  system  actors  to  ensure  these  reforms  would  be  put  in  place.  On  the  whole, 
the  outlook  for  how  these  reforms  should  impact  on  these  services  appears  to  be  quite 
positive. 

The  strategies  to  incorporate  buy-in  were  particularly  notable  in  this  case,  and  may 
offer  important  benefit  two  other  service  systems  engaged  in  resource  allocation  exercises.  In 
particular,  using  senior  management  to  validate  allocation  weights  and  variables,  using 
reliable  data  collected  and  reported  by  a  widely  recognized  advocacy  organization,  and 
reviewing  methods  publicly  and  repeatedly  with  contract  agencies  reduced  resistance  to 
funding  reallocations.  In  addition,  qualitative  vahdation  of  the  variables  and  weights  used  in 
this  formula  provide  methodological  support  in  a  situation  where  the  metric  of  interest, 
preventing  foster  care  placement,  is  inherently  unmeasurable. 

This  case  may  offer  generalizable  lessons  for  public  agency  managers  concerned  with 
issues  of  resource  distribution.  One  issue  made  clear  through  this  work  is  the  need  for  senior 
managers  and  policy  advocates  to  consider  issues  of  equity  in  the  distribution  of  program 
resources.  Even  relatively  simple  analytic  methodologies  can  be  used  to  undertake  periodic 
reviews  of  resource  distributions.  These  reviews  should  include  collection  and  reporting  of 
available  data  on  financial  expenditures,  chents  served,  and  per-client  budget  costs  across  any 
relevant  geographical  or  organizational  service  units  within  the  district.  Wherever  feasible, 
districts  may  also  wish  to  consider  collecting,  aggregating  and  reporting  available  population, 
demographic,  service  need  or  administrative  data  across  these  same  service  units.  These 
measures  can  provide  a  useful  comparison  against  current  resource  allocations,  identifying 
locations  or  service  units  where  adjustments  of  resource  allocations  may  be  desirable.  Even  if 
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major  adjustments  in  resource  allocations  are  not  contemplated,  the  ready  availability  of  such 
information  could  ensure  that  equity  issues  are  considered  as  part  of  routine  decision  making. 

A  second  important  element  in  this  work  is  its  reliance  on  clear  methodologies  that 
seek  to  remove  both  the  reality  and  appearance  of  bias  from  the  resource  allocation  decision, 
and  increase  validity  of  all  measures  used.  While  the  approaches  used  in  this  setting  were 
moderately  complex,  they  were  appropriate  to  the  scope  of  services  in  question.  Moreover, 
once  the  approaches  were  explained  to  providers  and  other  system  actors,  they  were  generally 
understood  and  perceived  as  significant  improvements  over  how  decisions  were  made  in  the 
past.  As  presented  in  Table  3,  the  specification  of  the  models  used  in  this  exercise  was 
important,  and  had  substantial  impacts  on  how  many  slots  were  assigned  in  each 
neighborhood. 

A  third  valuable  set  of  lessons  from  this  work  is  the  need  to  balance  analytic  results 
with  policy  and  programmatic  demands  of  the  services  in  question.  One  exemplar  of  this 
point  in  the  New  York  City  case  was  the  decision  to  limit  resource  reductions  in  any 
community  district  to  25  percent,  even  if  the  resource  allocation  methodology  suggested  that 
greater  reductions  were  merited.  While  these  limits  did  mean  that  some  communities  would 
continue  to  receive  more  resources  than  they  might  otherwise  be  entitled  to,  they  also  mitigate 
impacts  on  individual  long  established  programs  that  might  otherwise  have  suffered  crippling 
resource  reductions.  Such  cuts  could  have  undone  the  efforts  agencies  had  made  over  many 
years  to  establish  and  maintain  their  work  with  families  and  children,  undermining  some  of 
ACS’  strongest  programs.  This  decision  also  had  unmistakable  political  benefits,  ensuring 
communities  and  their  leaders  that  impacts  on  programs  would  have  some  limits. 
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Other  lessons  may  be  less  generalizable,  but  still  worthy  of  note.  Perhaps  the  most 
important  of  these  is  that  undertaking  these  reforms  during  a  period  of  substantial  expansion 
of  available  resources  clearly  made  this  work  more  readily  acceptable  to  all  concerned. 
However,  while  additional  resources  certainly  mitigate  negative  perceptions  of  these  changes 
in  affected  communities,  the  case  could  be  made  that  systematic  resource  allocations  are  most 
necessary  during  times  of  steady  or  declining  resources  when  each  available  dollar  must  be 
used  to  maximum  benefit  [Congressional  Budget  Office,  1980;  Gramlich,  1976].  Analytic 
approaches  that  seek  to  balance  resources  more  fairly  across  need  groups  can  best  ensure  the 
maximum  marginal  return  on  available  resources,  maximizing  the  total  benefit  of  these 
resources. 

Finally,  it  should  be  noted  that  the  specific  methodologies  described  in  this  paper  are 
not  the  only  ones  available.  In  many  cases,  much  simpler  approaches  may  be  used  quite 
effectively,  especially  in  situations  where  total  available  resources  are  smaller,  or  where  the 
geographic,  demographic  and  programmatic  service  distribution  is  less  diverse  than  was  the 
case  in  New  York  City.  On  the  other  hand,  other  services  or  service  systems  may  be  much 
more  complex,  and  will  no  doubt  require  more  intricate  and/or  more  precise  anal3^c 
approaches  which  can  increase  confidence  in  the  validity  as  compared  to  those  outlined  here. 
The  central  item  of  concern  should  be  to  develop  data  sources  that  are  valid,  reliable  and 
consistent,  to  use  methodologies  that  are  clear,  unbiased  and  appropriate  to  the  analytic  task  at 
hand,  and  to  seek  to  communicate  the  overall  approach  and  its  results  in  a  manner  understood 
by  all  relevant  actors. 
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Chapter  5.  Matching  Needs  and  Services:  an  Outcome-Based  Needs  Assessment  Tool 
for  Community-Based  Service  Systems 

Introduction 

Child  and  family  service  systems  across  the  United  States  are  increasingly  focusing  on 
the  community  context  of  their  services  [Robertson  &  Wier  1998].  These  include  national 
reform  models  such  as  Family-to-Family  [Annie  E.  Casey  1999]  and  Community 
Partnerships  for  Child  Protection  [Edna  McConnell  Clark  Foundation,  no  date],  as  well  as 
community-based  services  reforms  in  Los  Angeles  and  New  York  City,  the  nation’s  two 
largest  districts.  While  these  reforms  employ  different  strategies  and  focus  on  different 
aspects  of  case  practice,  they  all  focus  on  enhancing  the  important  roles  that  community 
residents  and  community-based  service  providers  play  in  addressing  the  needs  of  vulnerable 
children  and  their  families. 

Community-based  service  reforms  require  new  sources  of  information  [Connell  et  al. 
1995].  Such  information  is  vital  to  ensure  that  responsibilities  devolved  to  the  community 
level  are  met  and  that  no  child  or  family  “slips  through  the  cracks”  in  the  system.  Data  also 
provide  system  leaders  with  the  tools  necessary  to  set  directions  and  ensure  the  fair 
distribution  of  resources  to  all  parts  of  these  systems.  While  the  child  and  family  service 
research  literature  is  increasingly  focusing  on  the  power  of  administrative  data  to  support 
these  needs,  these  data  are  only  a  starting  point  for  understanding  the  complex  and  djmamic 
needs  of  the  children  and  families  who  typically  come  to  the  attention  of  child  welfare 
authorities  [Wulczyn  et  al.  2000;  Wulczyn  et  al.  1997].  Supplementing  administrative  data 
with  qualitative  information  provides  a  more  “three-dimensional”  portrait  of  the  challenges 
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facing  families  and  of  the  service  strategies  that  might  better  meet  these  challenges  [Usher  et 
al.  1995;  Altshuler  &  Gleeson  1999]. 

Qualitative  data  can  be  collected  using  case  record  reviews,  surveys,  focus  groups,  or 
structured  assessment  tools  derived  from  the  social  and  behavioral  sciences.  Collecting  valid 
qualitative  data  on  the  multidimensional  and  changing  issues  facing  at-risk  children  and  their 
families  is  difficult  and  costly,  requiring  careful  attention  to  methodology.  This  can  reduce 
the  scope  of  what  can  be  reliably  examined.  Even  when  high-quality  qualitative  information 
is  collected,  it  can  be  difficult  to  communicate  it  to  those  who  most  need  it — line  staff  and 
clients.  Finally,  these  approaches  rarely  reflect  the  voice  and  perspectives  of  line  staff, 
managers,  parents,  children,  and  others  who  have  substantial  day-to-day  experience  in  how 
child  welfare  services  work  and  the  needs  of  children  and  families  served  by  these  systems. 

The  Matching  Needs  and  Services  (MNS)  approach  builds  upon  traditional  qualitative 
research  techniques,  supporting  the  collection  of  high-quality  information  on  the  service 
needs  of  vulnerable  children  and  their  families.  It  explicitly  includes  the  voice  and 
perspectives  of  those  who  work  in  and  are  affected  by  these  systems  on  a  day-to-day  basis. 
Developed  by  the  Dartington  Social  Research  Unit  in  England,  MNS  has  been  used  in  many 
local  districts  in  the  United  Kingdom,  Norway,  Italy,  Spain,  and  the  United  States  [Dartington 
1999].  MNS  uses  a  relatively  simple  case  review  methodology  in  which  small  teams  of 
participants  from  social  service  systems,  along  with  individuals  served  by  these  systems, 
identify  the  primary  service  needs  of  a  defined  target  population  and  formulate 
recommendations  for  best  meeting  these  identified  service  needs. 

This  article  presents  an  overview  of  the  MNS  methodology,  as  well  as  a  detailed  case 
example  from  the  implementation  of  this  approach  by  the  Administration  for  Children’s 
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Services  (ACS)  in  New  York  City,  which  is  using  the  MNS  methodology  as  a  central 
component  of  its  newly  implemented  Neighborhood  Based  Services  strategy  [Scoppetta 
1996].  Also  presented  are  results  from  ACS’  pilot  review  using  the  MNS  methodology,  as 
well  as  an  examination  of  the  issues  involved  in  moving  toward  implementing  strategies 
emerging  from  this  analysis.  Finally,  recommendations  for  how  this  approach  may  be  used  in 
a  variety  of  social,  health  and  education  service  systems  are  offered. 

Methodology 

As  noted  above,  MNS  is  a  planning  method  that  seeks  to  clarify  the  patterns  of  need 
among  children  and  families  served  by  a  provider  agency,  using  statistically  valid  sample 
designs  and  rigorous  methodology.  It  is  explicitly  assumed  that  the  MNS  review  will  inform 
the  structure  and  operation  of  the  services  delivered  to  clients  and  that  incorporating  the  voice, 
experience  and  perspective  of  staff  and  clients  directly  involved  in  these  systems  will  yield 
more  effective  analysis  and  more  useful  data  to  support  service  enhancements. 

The  eight  elements  of  the  MNS  process  are  described  below,  with  estimates  of  the 
time  required  to  complete  each  step  in  a  given  review  “cycle.”  The  following  paragraphs 
detail  the  stages  of  work  involved  in  completing  one  MNS  review  cycle.  These  descriptions 
include  estimates  of  the  amount  of  time  required  to  complete  each  stage  in  the  New  York  City 
case.  Total  time  to  complete  all  stages  in  an  MNS  analysis  will  include  periods  where  work  is 
taking  place  on  more  than  one  stage  at  a  time,  as  well  as  gaps  in  time  between  the  end  of  one 
stage  and  the  beginning  of  another.  Dartington  staff  beheved  this  process  could  be  completed 
in  as  little  as  six  weeks,  but  it  required  more  than  six  months  total  elapsed  time  in  the  New 
York  City  case  [Dartington  1999]. 
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1.  Study  Design  and  Preparation  (4-6  weeks):  The  MNS  process  starts  with  a 
commitment  from  executive  staff  to  dedicate  the  resources,  time,  and  organizational  focus 
needed  by  this  project.  This  process  requires  a  project  manager  and  group  of  reviewers  who 
will  dedicate  significant  time  to  this  analysis.  The  reviewers  are  eight  to  twelve  professionals 
with  a  mix  of  experience  in  the  service  system,  including  line  caseworkers  and  supervisors, 
program  development  staff,  researchers,  senior  managers,  and  colleagues  from  cooperating 
service  systems.  Reviewers  will  be  required  to  make  a  four-  to  six-day  time  commitment  to 
complete  their  work  in  this  process,  although  this  can  be  spread  over  several  weeks  if 
necessary. 

2.  Sampling  Frame  and  Case  Record  Collection  (3-4  weeks):  The  MNS  process  starts 
with  identifying  and  gathering  a  representative  sample  of  100  to  500  case  files  from  the 
service  area  being  examined.  Cases  can  be  drawn  from  almost  any  portion  of  the  service 
system,  including  protective  services,  preventive  services,  or  foster  care  placement  cases,  or 
from  a  combination  of  these  elements.  The  MNS  review  tools  have  been  developed  explicitly 
within  the  context  of  child  and  family  services,  and  all  information  needed  should  be  available 
within  standard  case  records  used  in  most  service  agencies.  Obviously,  all  legal,  regulatory 
and  case  practice  issues  surrounding  confidentiality  of  case  records  must  be  observed.  In 
practice,  however,  the  most  substantial  challenge  in  this  stage  lies  in  actual  collection  and,  if 
necessary,  photocopying  of  the  case  records. 

The  key  feature  of  this  sample  selection  is  that  it  is  longitudinal,  not  cross-sectional 
[Darlington  1999].  The  recommended  sampling  design  is  to  select  cases  sequentially  as  they 
enter  the  service  system  from  a  given  starting  date.  This  ensures  the  most  representative 
sample  possible,  avoiding  biases  that  can  result  from  cross-sectional  samples  of  “in  care” 
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populations  that  tend  to  over-weight  cases  that  receive  service  for  longer  periods.  Sequential 
sampling  also  allows  more  accurate  measurement  of  the  number  of  clients  expected  to  enter  a 
need  group  over  a  given  period  of  time. 

3.  Review  Team  Training;  Record  Review;  Audit  Forms  (1-2  days);  After  the  group  is 
provided  a  background  briefing  on  the  MNS  process  and  how  the  agency  expects  to  use  the 
information  gathered  by  this  review,  training  is  conducted  during  each  stage  of  the  review 
cycle,  typically  by  having  the  group  work  together  to  complete  one  or  two  example  cases 
[Dartington  website:  http:\\www.dartington-i.org1.  It  is  especially  important  to  familiarize  all 
reviewers  with  the  layout  of  case  records  and  where  they  can  obtain  information  needed  for 
different  sections  of  the  review  form. 

Case  record  reviews  focus  on  six  domains  of  child  and  family  life:  living  situation; 
family  and  social  relationships;  physical  and  psychological  health;  education  and 
employment;  social  and  antisocial  behavior;  and  other  issues,  including  financial  status.  A 
two-page  review  form  is  designed  to  protect  client  confidentiality  and  can  be  modified  to 
reflect  the  circumstances  of  a  given  analysis  in  each  system.  Reviewers  are  directed  to  keep 
information  brief,  to  the  point,  and  in  laymen’s  terms.  This  format  proved  highly  effective  in 
collapsing  the  often-daunting  amount  of  data  in  a  typical  service  record  into  a  highly 
compacted  space  that  was  useful  for  later  stages  of  the  MNS  analysis. 

The  MNS  review  tool  examines  the  six  domains  from  three  perspectives,  each 
reflected  in  an  individual  column  of  the  form.  Reviewers  first  focus  on  the  child  and  family 
situation  on  the  day  they  began  service,  across  the  six  domains.  Second,  reviewers  must 
identify,  in  straightforward  language,  what  the  needs  are  for  the  child  and  family  in  each  of 
the  six  domains  at  point  of  entering  services.  This  is  a  critical  step,  as  it  forces  the  reviewer  to 
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think  in  terms  of  needs,  as  opposed  to  services.  This  shift  in  thinking — away  from  a  focus  on 
services  and  onto  the  underlying  family  and  child  needs — is  one  of  the  biggest  hurdles  for 
reviewers  to  overcome  in  the  MNS  process.  The  reviewer  must  focus  on  what  concrete  action 
needs  to  occur,  and  how  this  action  would  be  observed.  The  challenge  here  is  to  adequately 
reflect  the  multifaceted  nature  of  challenges  facing  at-risk  families,  in  language  that  is  clear, 
jargon-free,  and  without  prematurely  moving  to  identify  the  services  that  might  be 
traditionally  used  to  meet  these  needs. 

The  third  column  of  the  form  asks  reviewers  to  establish  desired  outcomes  for  the 
child  and  family  in  each  domain.  This  information  will  be  of  great  importance  during  the 
service-design  phase,  as  new  services  will  be  based  on  the  desired  outcomes  for  all  children 
identified  as  part  of  each  need  group.  It  is  imperative  that  the  outcomes  capture  the 
complexities  of  the  child  and  families’  problems.  Lastly,  reviewers  collect  data  on  the  actual 
service  experiences  of  the  child  and  family  and  enter  the  date  on  the  reverse  side  of  the  form. 

Using  a  team  of  eight  to  12  reviewers,  the  process  of  completing  100  review  forms 
takes  approximately  one  to  one  and  one-half  days,  with  each  record  requiring  approximately 
10  to  15  minutes  to  be  reviewed.  At  the  end  of  this  work,  information  from  tens  of  thousands 
of  pages  of  case  records  has  been  summarized  in  a  consistent  manner  on  a  single  two-page 
form  for  each  child  included  in  the  sample.  These  forms  provide  the  source  data  for  the 
remainder  of  the  MNS  review  exercise. 

4.  Sort  Forms  into  Need  Groups  (1  day):  Once  the  forms  have  been  completed,  the 
team  engages  in  the  sorting  session,  which  is  in  many  respects  the  centerpiece  of  the  MNS 
methodology.  Each  member  of  the  review  team,  again  drawing  on  his  or  her  own  unique 
training  and  experience,  makes  a  determination  as  to  the  primary  need  for  the  children  and 
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families  in  each  service  case.  Based  on  that  determination,  the  review  forms  are  sorted  into 
need  groups  reflecting  these  primary  needs.  Some  time  is  allowed  for  debate  and  discussion, 
but  the  individual  who  completed  the  form  makes  the  final  determination.  After  all  cases  are 
sorted,  the  results  are  reviewed  to  allow  for  the  possibility  of  reshuffling  of  particular  cases. 
The  piles  created  by  this  shuffling  exercise  represent  the  need  groups  identified  by  the  MNS 
process.  Typically,  eight  to  12  need  groups  are  identified. 

5.  Define  Need  Groups;  Establish  Desired/Expected  Outcomes  ( 1  day):  Once  the 
shuffling  process  produces  the  initial  sorting  of  cases  into  need  groups,  the  review  team  works 
to  create  clear  definitions  for  each  identified  need  group  and  establish  desired  outcomes  for 
any  child  and  family  included  in  each  need  group.  Again,  this  is  an  opportunity  to  accurately 
capture  the  complex,  multifaceted  nature  of  service  needs,  allowing  reviewers  to  lay  the 
groundwork  for  designing  services. 

To  define  and  establish  outcomes  for  each  need  group,  reviewers  focus  on  the  multiple 
issues  impacting  particular  groups  of  families  within  particular  communities.  It  has  long  been 
recognized  that  the  challenges  facing  at-risk  children  and  their  families  are  often  multiple  and 
can  change  over  time  (Halfon  et  al.  1994;  Koegel,  et  al.  1995].  Although  each  child  and 
family  situation  is  in  many  respects  unique,  the  MNS  method  seeks  to  use  case-level  data  and 
the  expertise  of  the  review  team  to  identify  common  patterns  of  need  that  underlie  these 
complex  and  dynamic  issues.  This  allows  information  from  individual  cases  to  inform 
analysis,  supporting  system,  and  community-level  reforms  at  a  more  fundamental  level.  This 
conversation  will  often  lead  the  group  to  further  refine  and  revise  the  initial  need  group 
sorting. 
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6.  Develop  Service  Design  Based  on  Outcomes  (1  day):  The  last  objective  of  the 
review  team  is  to  formulate  service  strategies  that  can  meet  the  needs  of  children  and  families 
within  each  need  group.  The  group  must  be  realistic  in  designing  implementable  service 
strategies  (“stay  on  the  page”),  but  not  be  constrained  by  thinking  only  of  current  service 
approaches  (“get  out  of  the  box”).  The  group  must  focus  at  all  times  on  the  actionable  needs 
presented  by  the  child  and  her  family.  At  this  stage,  service  strategies  are  only  described  at  a 
conceptual  level,  with  less  focus  on  operational  program  details.  Instead,  the  group  tries  to 
reach  consensus  on  service  strategies  that  it  believes  can  most  effectively  and  efficiently 
address  the  primary  service  needs  facing  families  in  each  need  group. 

7.  Design  and  Implement  Services  at  the  Community  Level  (6-8  weeks):  Once  the 
review  team  completes  its  work,  MNS  becomes  similar  to  a  traditional  planning  process.  The 
MNS  authors  recommend  that  this  be  undertaken  by  dedicated  service  planning  staff 
[Darlington  1999].  Planners  should,  however,  hold  routine  briefings  with  the  review  team  as 
the  work  of  program  design  and  implementation  unfolds. 

Deciding  which  service  options  to  pursue  is  a  key  question  at  this  stage.  The  choice  of 
service  priorities  requires  significant  judgment  on  the  part  of  planners  and  senior  managers, 
and  is  driven  by  the  individual  context  of  the  service  agency.  Several  criteria  are  available  to 
focus  this  decision;  the  largest  group;  the  most  widely  perceived  need  among  agency  leaders 
and  community  actors;  areas  with  resources  available  to  support  new  program  designs; 
service  strategies  most  ready  to  implement  in  the  service  organization;  or  those  groups  which 
have  historically  received  less  focus  and  attention  from  the  service  agency  [Dartington  1999]. 

8.  Evaluate  Service  Effectiveness  v^.  Desired  Outcomes  (4-6  weeks):  A  final  stage  in 

the  MNS  process  is  to  evaluate  the  services  that  are  ultimately  implemented.  One  key 
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advantage  in  MNS  is  that  the  relatively  clear  exposition  of  need  group  definitions  and  desired 
outcomes  can  greatly  facilitate  well  targeted  data  collection  and  less  ambiguous  interpretation 
of  evaluation  data. 

Analytic  Design 

In  1998,  New  York  City’s  Administration  for  Children’s  Services  (ACS)  identified 
MNS  as  a  potentially  useful  tool  in  its  efforts  to  design  and  implement  neighborhood-based 
child  and  family  services.  ACS  chose  to  pilot  MNS  using  five  separate  review  “cycles,” 
examining  a  total  of  333  children.  Each  cycle  reviewed  between  83  and  127  records,  using 
two  case  samples  from  the  South  Bronx  and  one  from  Staten  Island.  Two  review  cycles  were 
staffed  by  ACS  management  and  planning  staff,  one  cycle  included  ACS  casework  staff,  and 
two  cycles  provided  client  validation — one  with  parents,  foster  parents,  and  parent  advocates, 
and  a  second  with  adolescent  foster  children.  This  pilot  was  designed  to  support  both 
aggregation  of  findings  across  all  five  cycles  and  to  offer  comparisons  across  geographic 
areas  and  review  teams. 

The  client  validation  cycles  required  two  days  to  complete  rather  than  five,  since 
parents  and  foster  children  started  with  already  completed  forms  from  previous  cycles  and 
focused  on  sorting  cases  and  identifying  service  strategies  for  each  group.  The  parent  review 
cycle  included  foster  parents  and  birth  parents  of  children  who  had  experienced  foster  care 
placement,  while  the  foster  child  reviews  were  conducted  by  adolescent  foster  children  aged 
14  and  above.  Client  validation  cycles  sought  to  provide  those  most  directly  impacted  by 
child  and  family  services,  children  and  their  parents,  the  opportunity  to  review  the  summary 
data  on  the  sample  cases  and  offer  their  own  case  sorting,  need  group  definitions,  and 
recommended  service  strategies.  It  was  hoped  that  these  responses  would  offer  a  distinct 
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voice  either  validating  or  contrasting  the  findings  from  the  reviews  conducted  by  ACS  staff, 
particularly  with  respect  to  design  and  operation  of  services  to  meet  identified  needs. 
RESULTS 

The  five  review  teams  identified  between  eight  and  1 1  different  need  groups.  Table  1 
presents  a  composite  of  all  eleven  identified  needs  groups,  need  definitions,  and  the  number 
of  cases  in  each  group  aggregated  across  the  five  review  cycles.  Substance  Abuse  Treatment, 
Permanency  through  Adoption/Custody,  and  Parenting  Skills  were  the  three  largest  identified 
needs,  representing  63  percent  of  all  cases.  However,  the  five  smallest  need  groups — 
Temporary  Care/Immediate  Reunification,  Practical  Help  with  Special  Needs,  Domestic 
Violence,  Independent  Living,  and  High  Risk/Multiple  Need  reflected  13  percent  of  cases 
sampled. 
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Table  1.  Composite  need  groups  and  definitions  across  five  MNS  Cycles. 


Need  Group 

Need  Group  Description 

Number 
of  Cases 

Percent 

1.  Housing 

Family  needs  safe,  affordable  housing,  and  comprehensive 
housing  advocacy. 

37 

7% 

2.  Parenting 
Skills 

Caregivers  must  meet  the  basic  needs  of  their  children  and  learn 
appropriate  methods  of  discipline.  Caregivers  also  need  to 
understand  normal  child/adolescent  development  and  how  to 
engage  their  child  in  age-appropriate  activities. 

94 

19% 

3.  Cope  with 
Trauma  and 
Loss/  Stable 
Environment 

Child  needs  a  stable  environment  where  he  can  address  issues 
associated  with  severe  trauma  and  loss.  Comprehensive 
substance  abuse  treatment,  and  physical  and  mental  health 
services  are  needed. 

23 

5% 

4.  Emotional 
Well-Being 

Children  and  parents  need  mental  health  services  to  keep  the 
family  united.  Parents  of  children  with  mental  illnesses  need 
training  and  respite  care. 

61 

12% 

5.  Temporary 
Care/ 

Immediate 

Reunification 

Children  need  temporary  care  as  a  result  of  caregiver’s  absence 
due  to  incarceration  or  hospitalization.  Family  should  use 
preventive  services  to  avoid  inappropriate  foster  care  placement. 

6 

1% 

6.  Practical 

Help  with 

Special  Needs 

Child  and  caregiver  need  education  and  skill  building  to  manage 
diagnosed  physical  and/or  behavioral  conditions. 

14 

3% 

7.  Domestic 
Violence 

Caregiver  must  provide  a  safe  and  violence-free  environment  for 
her  children  and  learn  how  to  become  independent  and  self- 
confident.  The  caregiver  also  needs  support  services  to  address 
substance  abuse  and  legal  problems. 

18 

4% 

8.  Substance 
Abuse 

Treatment 

Parents  and  children  in  this  group  must  cease  abusing  drugs  and 
establish  a  drug-free  home.  Parents  need  to  learn  appropriate 
ways  of  caring  for  children.  Older  children  must  understand  the 
effect  drug  addiction  has  on  their  parent’s  ability  to  parent. 

Some  clients  may  need  to  detoxify  or  receive  HIV/medical  care. 

114 

23% 

9.  Permanency 
Through 
Adoption/ 
Custody 

Children  need  a  permanent,  safe,  stable  home  through  either 
adoption  or  custody.  Reunification  with  the  birth  father/mother 
is  not  advisable  or  impossible.  Children  need  appropriate 
mental  health  services  to  assist  in  adjusting  to  their  new  living 
situation. 

106 

21% 

10.  Permanency 
Through 
Independent 
Living 

Children  need  alternate  living  arrangements  due  to  an  inability 
to  return  to  natural  caregiver.  Children  must  complete  their 
education  and  develop  skills  in  personal  and  health  care, 
housing,  employment,  vocational  training,  and  financial 
management. 

7 

1% 

11.  High 

Risk/Multiple 

Need 

The  child  must  be  removed  from  harm’s  way  while  addressing 
multiple  issues  underlying  the  unsafe  situation.  This  need  group 
has  multiple  severe  needs  that  weigh  equally  in  preventing 
reunification,  rather  than  a  single  primary  definable  need. 

21 

4% 

Total 

501* 

*Note:  These  5( 

01  observations  reflect  333  individual  cases. 
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The  five  review  cycles  revealed  many  differences  and  similarities,  as  presented  in 
table  2,  below.  Three  need  groups  were  identified  in  every  cycle — ^Housing,  Parenting  Skills, 
and  Substance  Abuse  Treatment.  Two  other  need  categories  were  identified  in  a  significant 
share  of  cases  — ^Permanency  through  Adoption  (21  percent  of  cases  overall)  and  Emotional 
Well-Being  (12  percent) — ^but  were  not  identified  by  participants  in  one  or  more  cycles.  The 
remaining  need  groups  included  between  1%  and  5%  of  the  total  sample  and  were  identified 
by  between  one  and  three  review  teams.  Only  one  need  group  was  identified  in  only  one 
cycle— High  Risk/Multiple  Needs  in  16  percent  of  the  Cycle  1  cases  reviewed. 


Table  2.  Percentage  of  Cases  Assigned  to  Need  Groups  by  Review  Cycle,  ACS  Matching 
Needs  and  Services  Analysis.  _ _ _ _ 


Review  Cycle: 

1 

2 

3 

4 

5 

Conununity: 

South 
Bronx  1 

South 
Bronx  2 

South 
Bronx  1 

South 
Bronx  1 

Staten 

Island 

Sample  Size: 

501 

101* 

127 

83* 

83* 

100 

Review  Team: 

Need  Group: 

Total 

ACS 

Policy 

ACS 
Line  Staff 

Parents 

Foster 

Children 

ACS 

Policy 

1.  Housing 

7% 

4% 

1% 

12% 

14% 

24% 

2.  Parenting  Skills 

19% 

13% 

18% 

24% 

29% 

2% 

3.  Stable  Environment/Cope 
with  Trauma  and  Loss 

5% 

8% 

— 

14% 

““ 

4.  Emotional  Well  Being 

12% 

13% 

18% 

— 

18% 

4% 

5.  Temporary 
Care/Immediate 

Reunification 

1% 

1% 

3% 

1% 

6.  Practical  Help  with 

Special  Needs 

3% 

7% 

— 

6% 

1% 

7.  Address  Domestic 
Violence 

4% 

12% 

— 

— 

““ 

6% 

8.  Substance  Abuse 

23% 

27%  1 

37% 

44% 

8% 

11% 

9.  Permanency  Through 
Adoption/Custody 

21% 

—  1 

21% 

30% 

47% 

10.  Permanency  Through 
Independent  Living 

1% 

2% 

— 

5% 

11.  High  Risk/Multiple 

Need 

4% 

16% 

— 

— 

■■ 

*  Note:  The  same  cases  were  used  in  Cycles  1,  3  and  4. 
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However,  Table  2  also  makes  clear  that  there  were  significant  differences  in  need 
group  assignments  across  these  five  review  cycles,  and  comparing  results  across  five  review 
cycles  and  three  samples  reveals  few  clear  patterns.  One  comparison  worthy  of  note  is 
between  Cycles  1  and  2,  which  both  reviewed  cases  from  the  South  Bronx.  Although  both 
panels  identified  Substance  Abuse  Treatment  as  the  largest  need  group,  ACS  line  casework 
staff  in  Cycle  2  assigned  a  higher  percentage  to  this  group  (37  versus  27  percent).  In  addition, 
line  staff  assigned  23  percent  of  cases  to  the  Permanency  through  Adoption  and  Independent 
Living  need  groups,  while  policy  and  planning  staff  in  Cycle  1  assigned  no  cases  to  these 
groups.  Conversely,  Cycle  1  assigned  12  percent  of  cases  to  the  Domestic  Violence  and  16 
percent  to  the  High-Risk  need  groups,  where  none  were  assigned  in  Cycle  2. 

Another  set  of  patterns  of  interest  is  geographic.  Policy  staff  reviewing  Staten  Island 
cases  in  Cycle  5  appeared  to  find  more  concrete  needs  among  cases  reviewed,  assigning 
almost  half  of  cases  to  the  Adoption  need  group  and  almost  one  quarter  to  Housing.  At  the 
same  time,  this  review  identified  only  7%  of  cases  falling  into  need  groups  associated  with 
clinical  or  therapeutic  needs  (Parenting  Skills,  Trauma  and  Loss,  Emotional  Well-Being,  and 
Special  Needs).  In  comparison.  Cycles  1  and  2  assigned  fewer  cases  to  Adoption  and 
Housing  groups,  but  38  percent  of  cases  to  the  therapeutic  categories. 

Finally,  the  client  validation  analyses  in  Cycles  3  and  4  used  cases  from  Cycle  1 . 
Fifty-six  percent  of  these  cases  received  matching  need  group  assignments  between  two  or 
three  of  these  teams.  While  the  parents  in  Cycle  3  assigned  44  percent  of  cases  to  Substance 
Abuse  Treatment,  the  foster  children  only  identified  this  need  for  8%  of  cases.  Both  groups 
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assigned  many  more  cases  to  the  Parenting  Skills  and  Housing  need  groups  than  were 
identified  by  ACS  staff  reviewing  South  Bronx  cases  in  Cycle  1 . 

DISCUSSION 

The  analysis  presented  here  has  substantial  limitations,  which  must  be  considered 
carefully  when  interpreting  the  results  of  this  pilot  project.  The  clearest  limitation  of  this 
process  is  revealed  in  the  substantial  differences  in  need  group  assignments  across  the  five 
separate  review  cycles,  which  would  appear  to  signal  challenges  in  consistency  and  perhaps 
reliability  of  this  analysis.  Though  this  pilot  study  sought  to  test  the  fullest  possible  range  of 
results  likely  to  emerge  from  this  methodology  by  intentionally  using  cases  from  a  variety  of 
Community  districts  and  included  a  wide  range  of  participants,  the  disparity  in  results  across 
these  cycles  remains  notable. 

Most  striking  may  be  the  differences  in  results  from  South  Bronx  cases  reviewed  by 
ACS  Policy  and  Line  Staff  in  Cycles  1  and  2,  respectively.  For  example,  while  Policy  Staff  in 
Cycle  1  found  approximately  one  in  12  cases  fell  within  the  Stable  Environment  and  Special 
Needs  groups,  one  in  eight  in  the  Domestic  Violence  group  and  one  in  six  in  High 
Risk/Multiple  Need  group.  Line  Staff  in  Cycle  2  failed  to  assign  any  cases  to  such  groups.  At 
the  same  time.  Line  Staff  found  one  in  four  cases  to  fall  in  the  adoption  need  group  versus 
none  in  Cycle  1,  and  also  assigned  notably  higher  shares  of  cases  to  the  Substance  Abuse, 
Emotional  Well  Being  and  Parenting  Skills  groups  than  did  the  Policy  team. 

However,  it  is  also  worth  noting  that,  with  the  exception  of  the  High  Risk/Multiple 
Need  group  identified  in  Cycle  1,  every  need  group  was  identified  by  two  or  more  review 
teams  during  the  pilot.  This  appears  to  demonstrate  some  degree  of  consistency  in  the  MNS 
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method’s  ability  to  identify  patterns  of  need  even  if  the  distribution  of  cases  assigned  to  need 
groups  will  vary. 

In  addition,  the  issue  of  reliability  and  consistency  of  results  may  have  to  be  balanced 
against  the  benefits  this  review  tool  offers  in  permitting  the  voices  and  participation  of  a  wide 
range  of  actors  in  the  planning  process.  In  a  sense,  including  these  participants  in  this  process 
may  increase  the  real  or  perceived  “validity”  of  these  findings.  In  the  language  of  qualitative 
research  methodology  this  might  be  considered  face  or  construct  validity,  arising  from  the 
reliance  on  consensus  methods  used  here.  In  addition,  this  method  may  offer  “political” 
validity,  in  terms  of  organizational  buy-in  and  political  support  for  the  design  and 
development  of  new  service  strategies  associated  with  such  consensus  measures. 

In  fact,  future  researchers  may  be  able  to  draw  lessons  from  other  consensus  research 
methods  in  trying  to  improve  consistency  and  reliability  of  results  from  this  method,  including 
increasing  training  time  for  review  team  members;  developing  more  structured  prompts  and 
increasing  training  to  allow  process  facilitators  to  provide  more  explicit  guidance  to  review 
teams  during  the  need  group  assignment  process;  providing  more  explicit  feedback  to  the 
group  in  the  form  of  measures  of  agreement  or  disagreement  on  each  case  reviewed;  and 
possibly  to  requiring  somewhat  larger  sample  sizes  in  these  reviews.  Even  greater 
consistency  could  probably  be  achieved  if  review  teams  were  constituted  more  homogenously 
of  clinical  or  other  “experts”,  though  this  could  significantly  compromise  the  key  benefit  of 
this  process — ^its  broad  inclusion  of  diverse  voices  into  the  planning  process. 

Even  during  the  pilot  stages  described  in  this  analysis,  the  Matching  Needs  and 
Services  process  did  have  substantial  impact  within  the  New  York  City  Child  Welfare 
services  system.  When  the  five-cycle  analysis  was  completed,  results  were  presented  to 
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senior  managers  in  the  ACS  system,  and  continuing  efforts  are  now  focusing  in  two  areas. 
First,  ACS  has  decided  to  focus  on  the  Housing  need  group,  taking  several  steps  towards 
enhancing  programs  and  services  to  meet  the  needs  of  families  with  a  primary  need  for 
adequate,  stable  housing.  These  steps  include  creating  a  dedicated  housing  development  unit 
within  the  agency,  working  with  service  providers  to  enhance  services  to  families  with  unmet 
housing  needs,  and  initiating  partnerships  to  develop  new  housing  resources  specifically 
targeted  to  the  needs  of  families  within  the  child  welfare  system. 

ACS  is  also  expanding  use  of  MNS  at  the  community  level.  As  part  of  its  movement 
towards  community-based  services,  ACS  is  requiring  that  providers  create  neighborhood 
service  plans  and  has  established  MNS  as  the  preferred  method  for  this  work.  Currently,  all 
foster  care,  preventive  and  homemaking  service  providers  with  ACS  contracts  in  the  Bronx 
are  starting  the  MNS  process  in  conjunction  with  trained  ACS  technical  assistance  teams. 
ACS  expects  to  complete  multiple  MNS  review  cycles  in  virtually  all  neighborhoods  of  New 
York  City  over  the  next  year. 

As  these  efforts  unfold,  ACS  expects  other  community-based  systems,  including  the 
police.  Board  of  Education,  and  hospitals  to  join  this  process.  The  results  of  these  analyses 
are  expected  to  generate  significant  opportunities  to  integrate  existing  services  at  the 
community  level  and  to  identify  other  needed  service  programs.  Special  ACS  units  dedicated 
to  implementing  MNS  service  designs  are  being  assembled  to  help  ensure  a  smooth  transition 
in  the  introduction  of  new  services.  ACS  anticipates  the  impact  of  MNS  on  service  delivery 
to  the  children  and  families  in  New  York’s  child  welfare  system  to  grow  over  time  and  to  be  a 
fundamental  element  in  the  ongoing  creation  of  neighborhood-based  service  delivery  systems. 


Matching  Needs  and  Services 


66 


Despite  the  many  positive  aspects  revealed  by  this  project,  some  limitations  are  clear. 
One  issue  is  the  resource  demands  of  this  process.  The  reviews  at  ACS  were  significantly 
facilitated  through  a  grant  of  $25,000  provided  by  the  Annie  E.  Casey  Foundation.  This  grant 
allowed  ACS  to  contract  directly  with  the  Dartington  Social  Research  Unit,  allowing 
facilitators  from  England  to  travel  to  New  York  to  train  ACS  staff  as  facilitators  and  to  guide 
completion  of  the  first  review  cycle.  These  resources  also  allowed  a  team  of  ACS  personnel 
to  visit  local  authorities  in  England  that  had  used  the  MNS  process  in  the  past.  While  the 
Dartington  staff  made  significant  contributions  to  the  successful  completion  of  the  MNS  pilot 
project  in  New  York  City,  available  published  materials  could  allow  many  local  districts  to 
succeed  in  implementing  MNS  reviews  either  on  their  own  or  with  relatively  little  training, 
should  outside  resources  not  be  available. 

Identifying  staffing  resources  is  another  area  of  potential  challenge  in  implementing 
MNS  reviews.  Although  many  planning  processes  are  resource  intensive,  MNS  requires  the 
participation  of  eight  to  12  staff  over  a  four-  to  six-day  period,  plus  several  weeks  of 
additional  work  from  a  core  planning  group.  As  such,  completing  a  single  MNS  review  will 
typically  require  between  60  and  150  person-days  of  effort.  Time  demands  increase  if 
multiple  cycles  are  undertaken,  though  some  efficiencies  are  possible.  Since  most  of  this 
resource  commitment  arises  from  the  contribution  of  the  staff  who  participate  in  the  case 
review  portion  of  the  process,  this  burden  can  be  spread  over  a  larger  number  of  participants 
than  may  be  common  in  many  traditional  strategic  planning  exercises  [Mintzberg  1994]. 

Another  concern  with  this  process  surrounds  what  happens  after  the  completion  of  the 
review  cycle.  Staff  from  the  Dartington  Social  Research  Group  have  made  it  clear  that 
successful  implementation  of  new  programs  arising  from  MNS  has  been  rare  [Dartington 
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1999].  Although  the  problem  of  implementation  is  always  challenging  in  any  social  service 
context  and  cannot  be  specifically  ascribed  to  the  MNS  process,  the  relative  rarity  of 
successful  implementation  of  new  programs  and  services  under  this  process  is  clearly  an 
important  issue. 

Perhaps  the  most  significant  challenge  facing  local  districts  seeking  to  implement 
findings  from  reviews  such  as  MNS  is  that  the  needs  identified  among  clients  can  be  expected 
to  be  complex,  multiple,  and  dynamic.  These  client  needs  will  frequently  give  rise  to  service 
recommendations  that  require  new  resources  or  flexible  use  of  available  resources. 
Implementing  such  services  will  often  demand  support  and/or  collaboration  from  multiple 
service  systems.  However,  the  MNS  methodology  is  analytically  rigorous,  can  be  clearly 
explained  to  actors  in  other  systems,  and  produces  results  that  can  be  readily  understood  by 
wide  range  of  actors  across  service  systems.  Thus,  while  this  process  of  itself  will  not  solve 
complex  issues  involved  in  inter-organizational  collaboration,  the  MNS  process  can  be 
expected  to  provide  clear  statements  of  client  needs  and  recommended  service  strategies 
which  will  facilitate  communications  with  other  system  actors. 

The  ACS  experience  offers  several  suggestions  for  ensuring  successful 
implementation  of  MNS  reviews  in  community-based  service  settings: 

1)  Senior  managers  in  the  reviewing  jurisdiction  must  be  fully  aware  of  this  process.  Their 
commitment  to  seeing  this  review  through  full  implementation  must  be  explicit  and 
widely  recognized  by  others  in  the  organization. 

2)  Senior  managers  and  planners  must  acknowledge  from  the  beginning  that  implementation 
is  a  serious  challenge  in  this  process.  Staff  participating  in  this  process  must  explicitly 
identify  strategies  to  ensure  effective  implementation. 
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3)  The  supervising  agency  must  remain  flexible  and  responsive  in  addressing  concerns  of 
participating  agencies  regarding  sample  size,  confidentiality  of  records,  and  completion 
schedules. 

4)  Building  in  child,  parent,  and  line  staff  participation  broadens  the  range  of  perspectives 
included  in  the  review  process  and  gives  voice  to  those  who  have  the  most  impact  and  are 
most  impacted  by  these  services. 

5)  Careful  attention  should  be  paid  to  disseminating  findings  from  these  reviews.  In 
particular,  reports  arising  from  this  work  should  be  written  to  be  accessible  to  all  actors 
impacted  by  these  systems,  including  children,  families,  line  staff  and  advocates. 

6)  Finally,  planners  and  senior  managers  should  be  strategic  in  deciding  which  service 
strategies  are  implemented  and  in  what  order  these  strategies  are  pursued.  In  particular,  it 
is  probably  a  good  idea  to  start  with  approaches  that  are  most  readily  implementable  in  the 
beginning,  even  if  these  reflect  smaller  need  groups  or  groups  with  less  extreme 
presenting  problems. 

Despite  these  important  limitations,  MNS  appears  to  offer  significant  potential  as  a 
tool  policy  researchers  can  use  in  planning  for  child  and  family  services.  This  method  could 
be  readily  adapted  for  use  in  a  variety  of  health,  education,  and  community  service  programs. 
In  particular,  its  focus  on  methodologically  consistent  collection  of  data  from  actual  case 
records  and  its  explicit  inclusion  of  those  who  work  in  and  receive  services  from  these 
systems  in  this  formalized  planning  process  appears  relatively  unique.  These  strengths  can 
reinforce  each  other  in  the  MNS  process,  yielding  results  that  are  both  rigorous  and 
meaningful  to  those  who  must  participate  in  any  successful  changes  in  these  systems. 
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Chapter  6.  Effects  of  Community-Level  Needs  on  Foster  Care  Outcomes 

Introduction 

While  there  is  general  recognition  that  child  and  family  service  evaluation  must  rely 
on  multiple  research  methods,  to  date  the  most  robust  evaluations  have  used  event  history 
analysis  to  model  measurable  program  outcomes  in  a  variety  of  child  and  family  service 
settings  [Wulczyn,  Orlebeke,  and  Melamid,  2000;  Fraser,  Pecora,  and  Popuang,  1992].  These 
efforts  have  increased  recognition  of  the  importance  of  high  quality  evaluation  in  community 
service  initiatives,  and  also  raised  awareness  of  the  many  challenges  facing  evaluators  in  this 
area. 

Researchers,  practitioners  and  children’s  services  advocates  generally  acknowledge 
the  role  community  context  may  play  in  all  services  to  vulnerable  children  and  their  famihes 
[Schorr,  1997].  The  benefits  of  attending  to  community  needs  in  these  services  include:  less 
disruption  in  the  child’s  life;  improved  service  to  the  child’s  family,  especially  by  reducing 
barriers  to  visitation  while  the  child  remains  in  care;  and  support  for  service  networks  within 
communities  most  impacted  by  child  abuse  and  neglect  which  can  better  ensure  the  safety  of 
reunified  children  and  reduce  risk  of  replacement  in  foster  care  [Pecora,  Whittaker,  Malluccio, 
Barth,  and  Plotnick,  1992;  Fanshell  and  Shinn,  1978].  Ultimately,  the  boundaries  between 
child  protective  services  (including  foster  care)  and  community  are  substantially  permeable: 
more  than  half  of  children  entering  foster  care  in  eleven  states  studied  by  Wulczyn,  Bruner 
and  Goerge  [1999]  eventually  returned  to  their  families  and  home  communities,  while  almost 
one  in  four  discharged  children  returned  to  care. 
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The  impact  of  community-level  needs  on  outcomes  for  children  in  foster  care  is 
explored  by  building  upon  proportional  hazards  event  history  models  used  to  evaluate 
programs  operated  by  the  New  York  City  Administration  for  Children’s  Services  (ACS) 
[Wulczyn,  Orlebeke  and  Melamid,  2000].  These  models  were  developed  as  part  of  this 
agency’s  community-based  reform  initiative,  and  were  designed  to  measure  performance  of 
41  non-profit  foster  care  agencies  who  provide  service  to  approximately  30,000  children  at 
any  point  in  time  under  contract  with  ACS. 

These  evaluations  sought  to  identify  agencies  where  children  experienced  outcomes 
from  foster  care  that  were  significantly  above  or  below  “average”,  with  the  intention  of 
providing  supports  for  agency’s  experiencing  poorer  outcomes  and/or  to  shift  placements 
towards  better  performing  providers.  The  analysis  presented  here  augments  these  evaluation 
models,  by  assigning  a  community  need  score  from  each  of  New  York’s  59  community 
districts  to  each  child  whose  record  indicates  their  home  address  in  the  evaluation  database. 
Incorporating  community  need  into  these  models  may  offer  insight  into  both  policy-level  and 
methodological  issues  that  arise  in  the  implementation  and  evaluation  of  community-based 
foster  care  services.  At  the  policy  level,  this  analysis  will  seek  to  assess  whether  the  impact  of 
community  context  on  the  outcomes  children  experience  in  foster  care  is  measurable,  and 
whether  children  entering  care  from  more  challenged  communities  are  likely  to  experience 
poorer  outcomes  from  foster  care  services.  Methodologically,  this  analysis  explores  the  use 
of  these  community  need  scores  as  a  proxy  for  child  and  family  level  risk  adjustment  to  the 
original  evaluation  model. 

Despite  the  wide  recognition  of  the  importance  of  community  context  in  these 
services,  little  attention  has  been  paid  to  the  impact  of  community-level  need  characteristics 
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on  foster  care  outcomes:  evaluating  how  the  level  of  need  in  different  communities  may  affect 
outcomes  of  children  from  those  communities  who  experience  foster  care  placement.  This 
paper  uses  data  from  the  nation’s  largest  child  welfare  system  to  explore  methodological, 
programmatic  and  policy  issues  in  measuring  community  need  and  in  incorporating  such 
measures  into  evaluations  of  outcomes  in  foster  care  services. 

Methodology 

Event  History  Outcomes  Models:  As  noted  above,  this  analysis  builds  upon  foster 
care  outcomes  evaluation  models  developed  by  ACS  in  conjunction  with  the  Chapin  Hall 
Center  for  Children  at  the  University  of  Illinois  [Wulczyn,  Orlebeke,  and  Melamid,  2000]. 
These  Outcomes  models  rely  on  proportional  hazards  survival  analysis  statistical  techniques, 
which  are  a  standard  approach  for  measuring  time  until  an  event  occurs,  such  as  reunification 
from  foster  care,  re-entry  into  care  or  completion  of  adoption  activities.  Such  models  hold 
significant  advantages  over  conventional  linear  regression  models,  when  some  members  of 
the  sample  under  study  have  not  experienced  the  event  of  interest  by  the  time  data  are 
compiled  [Fraser,  Pecora,  and  Popuang,  1992].  Proportional  hazard  models  seek  to  calculate 
the  likelihood  of  a  specific  event  occurring  to  an  individual  with  a  given  set  of  characteristics, 
and  generates  relative  risk  ratios  for  dependent  variables  included  in  the  model.  Proportional 
hazard  models  allow  analysts  to  consider  "censored"  cases  which  have  not  experienced  an 
outcome  during  the  analysis  period  in  the  calculation  of  risk  likelihood  ratios. 

These  evaluation  models  use  administrative  data  from  the  ACS’  Child  Care  Review 
Service  (CCRS)  computer  database  to  estimate  length  of  time  for  children  to  reunify  after 
foster  care  placement,  re-enter  into  care  following  discharge,  or  to  have  a  finalized  adoption. 
The  models  use  demographic  and  child  placement  variables  available  in  the  CCRS  system  as 
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independent  variables,  including  child  gender,  race,  age  at  the  beginning  of  placement, 
financial  eligibility  (a  proxy  for  poverty),  year  of  admission,  presence  of  siblings  in  care,  type 
of  placement  service  (traditional  foster  care,  kinship  foster  care,  group  home,  institution,  etc.), 
and  previous  spells  in  care.  A  series  of  dummy  variables  representing  the  41  contract  foster 
care  agencies  providing  service  to  the  child  are  also  entered  into  the  model,  generating 

3 

estimates  of  the  impact  of  the  agency  on  child  outcomes  in  foster  care. 

Five  models  are  estimated  in  this  framework.  Two  models  estimate  reunification 
rates— one  for  family-based  care  (traditional  foster  boarding  homes  and  placements  with 
relatives),  the  second  for  congregate  care  (institutions,  group  homes,  group  residences,  agency 
boarding  homes,  and  other  specialized  care  settings).  Two  other  models  estimate  re-entry 
rates  for  family  based  and  congregate  care.  The  final  model  estimates  adoption  finalization 
rates  for  family-based  care.  Samples  are  drawn  representing  all  children  first  admitted  to  care 
from  1995  through  1997  for  Reunification,  and  1993  through  1995  for  Adoption  models.  The 
earlier  sampling  period  for  Adoption  is  designed  to  permit  the  many  legal  and  casework 
activities  required  to  be  completed  in  this  service  pathway.  Re-entry  samples  represent  the 
universe  of  children  discharged  from  their  first  spell  in  foster  care  from  1996  through  1998. 

Calculating  Community  Need  Scores:  In  the  analysis  presented  here  the  basic 
models  are  augmented,  adding  a  community  needs  score  reflecting  the  level  of  risk  for  each 
child  included  in  this  analysis  from  a  given  community.  Community  need  estimates  used  in 
this  analysis  were  adapted  from  community  allocation  scores  calculated  to  support  financial 
allocations  for  preventive  services  across  New  York  City's  59  community  districts  [Melamid, 

Under  New  York  city's  foster  care  system  approximately  80%  of  children  in  care  receive  service  from  nonprofit 
contract  foster  care  providers,  with  the  remaining  children  receiving  service  from  directly  managed  public 
service  units.  This  analysis  reflects  results  from  models  run  using  data  on  contract  providers  only. 
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2001].  The  development  and  content  of  these  need  scores  are  presented  in  Chapter  Four  of 
this  dissertation.  Table  1  presents  data  on  the  allocation  of  service  needs  across  New  York’s 
59  community  districts  [Melamid,  2001]. 

Linking  Community  and  Outcomes  Data:  Community  need  scores  were  linked  to 
outcomes  data  using  family  address  information  contained  in  the  CCRS  database.  Analysis  of 
missing  data  patterns  in  the  adoption  model  revealed  67.7%  of  cases  had  missing  address 
data.  These  missing  data  arose  from  two  factors.  First,  the  long  time  lines  associated  with 
children  moving  into  and  through  the  adoption  process  requires  that  the  outcome  models 
focus  on  children  admitted  into  care  as  much  as  eight  years  earlier.  Second,  and  more 
importantly,  children  freed  for  adoption  are  assigned  new  case  numbers  which  are  different 
from  their  family  case  number.  As  a  result,  the  adoption  model  was  not  able  to  be  included  in 
this  analysis  of  the  impacts  of  conununity  need  on  child  outcomes  in  foster  care. 
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Table  1.  Community  Need  Shares  for  59  New  York  City  Community  Districts,  from  Melamid 


[2001]. 


Community 

CD  Need 

Community 

CD  Need 

District 

Allocation 

District 

Allocation 

Bronx 

25.5%  Manhattan 

16.3% 

Mott  Haven 

XI 

2.7% 

Battery  Park 

Ml 

0.1% 

Hunts  Point 

X2 

1,4% 

Greenwich  Village 

M2 

0.3% 

Morrisania 

X3 

2.0% 

Lower  E.  Side 

M3 

2.4% 

Concourse 

X4 

3.6% 

Chelsea 

M4 

0.5% 

JJniv.  Heights 

X5 

3.7% 

Midtown 

M5 

0.1% 

E.  Tremont 

X6 

1.9% 

Murray  Hill 

M6 

0.3% 

Fordham 

X7 

2.5% 

Upper  W.  Side 

M7 

1.3% 

Riverdale 

X8 

0.8% 

Upper  E.  Side 

M8 

0.5% 

Soundview 

X9 

3.1% 

Manhattanville 

M9 

1.9% 

Throgs  Neck 

XIO 

0.9% 

Central  Harlem 

MIO 

2.6% 

Pelham  PkMj, 

XU 

1.1% 

E.  Harlem 

Mil 

3.0% 

Williamsbridge 

X12 

1.9% 

Washington  Hts. 

M12 

3.2% 

Brooklyn 

36.8% 

Queens 

17.8% 

Williamsburg 

K1 

2.6% 

Astoria 

Q1 

1.6% 

Fort  Greene 

K2 

1.5% 

Sunnyside 

Q2 

0.6% 

Bedford  Stuyvesant 

K3 

4.1% 

Jackson  Hts. 

Q3 

1.1% 

Bushwick 

K4 

2.9% 

Elmhurst 

Q4 

1.1% 

E,  New  York 

K5 

4.6% 

Ridgewood 

Q5 

1.2% 

Park  Slope 

K6 

1.2% 

Forest  Hills 

Q6 

0.4% 

Sunset  Park 

K7 

1.3% 

Flushing 

Q7 

1.0% 

N  Crown  Hts, 

K8 

2.3% 

Fresh  Meadows 

Q8 

0.9% 

S,  Crown  Hts, 

K9 

2.0% 

Woodhaven 

Q9 

1.1% 

Bay  Ridge 

KIO 

0.6% 

Howard  Beach 

QIO 

1.3% 

Bensonhurst 

Kll 

0.9% 

Bayside 

Qll 

0.5% 

Borough  Park 

K12 

1.5% 

Jamaica 

Q12 

3.7% 

Coney  Island 

K13 

1.6% 

Queens  Village 

Q13 

1.5% 

Flatbush 

K14 

2.2% 

Rockaways 

Q14 

1.8% 

Sheepshead 

KIS 

1.0% 

Staten  Island 

3.5% 

Brownsville 

K16 

3.0% 

Willowbrook 

SI 

2.0% 

E,  Flatbush 

K17 

2.2% 

South  Beach 

S2 

0.8% 

Canarsie 

K18 

1,4% 

Tottenville 

S3 

0.7% 

Unfortunately,  a  large  share  of  the  cases  used  in  the  other  four  models  also  had 
missing  address  data:  29.1%  in  the  Reunification  Family  sample,  47.4%  of  Reunification 
Congregate,  32.6%  of  Re-Entry  Family,  and  36.2%  of  Re-Entry  Congregate.  T-tests 
comparing  mean  values  of  independent  and  dependent  variables  revealed  that  patterns  of 
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missing  data  would  introduce  bias  into  this  analysis  if  left  unaddressed."^  Missing  data  biases 
were  corrected  through  propensity  weights,  estimated  by  regressing  a  flag  for  data  being  non¬ 
missing  on  a  large  number  of  independent  variables  using  logistic  regression  analysis.  These 
variables  included  child  duration  in  care,  reunification  and  re-entry  outcomes,  and 
demographic  and  child  placement  history  variables.  Initial  results  were  analyzed  to  refine  the 
fit  of  the  model  and  to  evaluate  for  outlier  cases  that  could  potentially  hold  inappropriate 
leverage  on  estimates  derived  fi'om  the  logistic  regression.^’^ 

Weights  were  calculated  from  these  logistic  regression  analyses  by  taking  the  inverse 
of  the  predicted  probability  of  having  non-missing  variables,  for  each  of  the  four  remaining 
models  to  be  included  in  this  analysis  [Hosmer  and  Lemeshow,  1989;  McCaffrey,  2001]. 
These  weights  were  entered  into  the  survival  analysis  models  using  Stata’s  probability 
weighting  option  [Stata  Press,  1997].  White/Huber  Sandwich  calculations  were  considered 
for  incorporation  in  the  model,  to  correct  calculations  of  significance  for  bias  which  would 
have  arisen  due  to  the  use  of  grouped  variables,  in  this  case  community  district  need  scores 
[Muthen  and  Masyn,  2001;  Stata  Press,  1997].  However,  these  calculations  were  not 
incorporated  in  these  analyses,  as  children  in  this  sample  represent  the  universe  of  children 
affected  by  these  services  at  this  time  period.  As  a  result,  levels  and  patterns  of  significance 


For  example,  the  mean  reunification  rate  for  children  with  missing  data  was  56.8%,  versus  52.3%  for  children 
whose  address  data  was  available,  t=5.80  P>r=0.00,  two  tailed.  Similar  results  occurred  for  other  Outcome 
measures  of  interest,  including  days  to  reunification  and  re-entry,  as  well  as  re-entry  rates. 

While  no  outlier  cases  were  excluded  from  this  analysis,  it  was  necessary  to  exclude  children  placed  with 
directly  managed  public  programs  from  the  family-based  service  analyses.  This  was  due  to  changes  in  data  entry 
practices  which  affected  the  consistency  of  service  agency  codings  during  the  period  of  this  analysis. 

^  Community  Needs  were  entered  into  the  model  as  a  single  continuous  variable.  While  the  analysis  could 
alternatively  have  used  the  seven  separate  need  scales  which  underlie  this  single  measure,  the  very  significant 
levels  of  correlation  across  these  measures  would  been  homoscedacistic,  producing  inaccurate  hazard  ratio 
calculations  for  these  variables.  In  addition,  one  of  these  scales.  Child  Safety,  is  comprised  of  measures  of 
abuse/neglect  and  foster  care  placement  rates  in  each  community,  and  would  have  been  significantly  identified 
with  the  outcome  measures  of  interest  in  this  analysis.  Finally,  introducing  seven  separate  scales  would  have 
reduced  the  efficiency  of  these  models. 
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reported  here  can  be  considered  unbiased  relative  to  experiences  within  this  system,  but  must 
be  considered  inflated  if  one  seeks  to  generalize  outside  the  system  under  study  here  [Bumam, 
2001]. 

RESULTS 

Descriptive  Statistics:  Table  2  presents  summary  statistics  on  the  children  included 
in  the  five  outcomes  models,  including  reunification  and  re-entry  samples  for  both  family  and 
congregate  care  and  family-based  adoption  cases.  Relatively  few  of  the  demographic  or 
placement  history  variables  exhibit  striking  differences  across  these  five  samples — children  in 
congregate  care  appear  to  be  substantially  older  than  those  in  family-based  care,  and  were  also 
much  less  likely  to  have  siblings  in  care.  Children  in  congregate  care  were  somewhat  less 
likely  to  be  reunified  with  their  families  (57  versus  65  percent),  and  more  likely  to  return  to 
care  if  they  are  reunified  (15  versus  9  percent).  Lastly,  child  outcomes  appear  to  vary  with 
community  needs:  children  in  the  most  needy  communities  (quartile  4)  appear  to  have 
somewhat  lower  reunification  rates  than  those  in  less  challenged  communities,  but  also  have 
lower  reentry  rates  once  reunified.  However,  children  with  missing  community  data  tend  to 
have  the  highest  reunification  rates,  and  the  lowest  re-entry  rates  of  any  group. 
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Table  2.  Mean  values  of  dependent  and  independent  variables  used  in  the  foster  care 
outcomes  analyses. 

Reunification  Models  Re-Entry  Models 
Family  Congregate  Family  Congregate  Adoption 


Observations 

17,344 

3,887 

13,284 

2,184 

14,821 

Female 

49% 

54% 

49% 

51% 

49% 

Placement  Age  <1 

23% 

2% 

19% 

1% 

26% 

Placement  Age  1 

9% 

0% 

9% 

1% 

8% 

Placement  Age  2-4 

23% 

1% 

22% 

1% 

21% 

Placement  Age  5-7 

18% 

1% 

20% 

1% 

20% 

Placement  Age  8-10 

12% 

2% 

15% 

4% 

13% 

Placement  Age  11-13 

10% 

21% 

10% 

21% 

8% 

Placement  Age  14-17 

5% 

73% 

4% 

72% 

4% 

Placement  Age  18+ 

0% 

0% 

0% 

0% 

0% 

Kinship  Foster  Care 

18% 

12% 

54% 

Institutional  Care  (25+  children) 

27% 

25% 

Group  Residence  (13-25  children) 

13% 

16% 

Group  Home  (7-12  children) 

51% 

50% 

Agency  Boarding  Home  (1-6  children) 

8% 

9% 

Residential  Treatment  Program 

0% 

— 

Eligibility-Federal 

47% 

35% 

30% 

33% 

81% 

Eligibility-Emergency 

42% 

46% 

58% 

54% 

7% 

Eligibility-Not  Ehgible 

11% 

18% 

12% 

13% 

12% 

Eligibility-Other 

0% 

0% 

0% 

0% 

0% 

Ehgibility  Missing 

0% 

0% 

0% 

0% 

0% 

White 

3% 

3% 

3% 

4% 

2% 

African  American 

42% 

39% 

39% 

28% 

56% 

Hispanic 

22% 

22% 

22% 

24% 

17% 

Other  Race 

3% 

3% 

3% 

2% 

1% 

Race  Unknown 

30% 

34% 

34% 

41% 

23% 

No  Siblings  Ever  in  Care 

23% 

68% 

23% 

64% 

55% 

Siblings  In  Care,  Not  Simultaneous 

3% 

9% 

3% 

8% 

2% 

Siblings  In  Care,  same  time 

74% 

23% 

74% 

28% 

43% 

Child  had  previous  placements 

11% 

29% 

9% 

21% 

69% 

Community  Need— Quartile  1 : 

17% 

14% 

17% 

16% 

9% 

Community  Need— Quartile  2: 

18% 

12% 

15% 

18% 

7% 

Community  Need— Quartile  3: 

19% 

12% 

18% 

12% 

8% 

Community  Need-Quartile  4: 

17% 

14% 

16% 

17% 

8% 

Community  Need— missing: 

29% 

47% 

33% 

36% 

68% 

Reunification  Rate 

Reentry  Rate 

Adopted 

Community  Need— Quartile  1: 

63% 

50% 

10% 

13% 

47% 

Community  Need— Quartile  2: 

59% 

55% 

12% 

22% 

46% 

Community  Need— Quartile  3: 

65% 

50% 

11% 

21% 

44% 

Community  Need— Quartile  4: 

59% 

46% 

9% 

14% 

48% 

Community  Need-missing: 

76% 

64% 

7% 

9% 

45% 

Overall  Average 

65% 

57% 

9% 

15% 

46% 

Days  to  Exit/Censoring 

463 

227 

363 

338 

1,176 

Community  Need  in  Outcomes  Evaluations 


78 


Outcomes  Analysis:  Table  3  summarizes  results  from  the  proportional  hazards 
analysis  models  for  reunification,  for  both  family-based  and  congregate  care  models.  For 
each  of  these  models,  results  are  presented  first  from  the  “base”  model.  These  results  reveal  a 
large  number  of  significant  relationships  between  the  control  variables  and  Reunification  from 
Family-Based  Care,  but  somewhat  fewer  patterns  of  significance  in  the  Congregate  model. 
Specifically,  the  Family-based  model  demonstrates  increased  likelihood  for  Reunification  for 
female  children,  all  age  categories  other  than  infants,  children  in  traditional  rather  than 
kinship  foster  care  settings,  those  who  are  not  eligible  for  Federal  reimbursement  (which 
serves  as  a  proxy  for  poverty  in  the  model),  those  admitted  in  1995,  those  who  from  the  white 
racial/ethnic  group,  and  those  who  do  not  experience  transfers  while  in  care.  Patterns  of 
significance  in  the  Congregate  sample  are  less  striking,  but  largely  follow  the  same  patterns  as 
the  Family-care  sample.  It  is  worthy  of  special  note  though,  that  the  Hazard  Ratio  for 
Reunification  of  children  experiencing  transfers  in  Congregate  care  is  0.48,  indicating  that 
experiencing  replacements  in  Congregate  care  may  reduce  Reunification  likelihood  by 
approximately  half. 

The  addition  of  community  needs  scores  to  the  basic  model  generally  has  limited 
effect  on  other  demographic  and  placement  experience  variables.  For  example,  while  gender 
is  significant  at  the  .01  level  in  the  basic  family  reunification  model,  it  is  not  significant  when 
community  needs  are  added  to  the  model.  Similarly,  while  both  admission  year  variables  are 
significant  at  the  .001  level  in  the  base  model  for  family  reunification,  admission  year  1996  is 
not  significantly  different  from  1995  admissions,  and  1997  admissions  are  only  significantly 
at  the  .10  level.  Among  children  in  congregate  care,  the  introduction  of  community  needs 
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scores  increases  the  hazard  ratios  for  all  ages  of  children,  with  significance  ranging  from  .05 
to  .001  levels,  and  increases  the  apparent  impact  of  admission  year  cohort  effects. 

Community  needs  scores  are  entered  into  the  model  using  two  methods.  When 
entered  as  a  continuous  variable,  with  a  score  for  each  child  based  on  the  needs  score  for  their 
home  community,  family-care  models  result  in  a  community  needs  hazards  ratio  below  1, 
indicating  that  increasing  community  needs  scores  are  associated  with  reduced  likelihood  of 
reunification  when  controlling  for  demographic  and  placement  history  variables.  However, 
the  Hazard  Ratio  for  congregate  care  is  1.04,  indicating  an  apparent  inverse  relationship 
between  community  needs  and  likelihood  of  Reunification  in  Congregate  Care.  This 
continuous  community  needs  variable  is  only  significant  at  the  .01  level  for  children  in 
family-based  care,  but  are  not  significant  for  children  in  congregate  care. 

Quartiles  of  community  need  are  also  entered  into  the  model,  to  identify  any  possible 
patterns  of  discontinuity  in  the  relationship  between  community  needs  and  child  reunification. 
While  the  congregate  care  analysis  again  reveals  no  significant  relationship,  analysis  of 
family-based  care  reveals  that  children  in  all  quartiles  of  communities  are  more  likely  to 
experience  reunification  than  those  in  the  most  needy  quartile  of  communities  (p<.01,  p<.10, 
and  p<.01,  respectively).  Complete  results  for  these  quartile  analyses  are  not  reported  here,  as 
they  closely  match  those  using  the  continuous  community  needs  variable. 

Lastly,  Table  3  summarizes  the  effects  of  service  agencies  on  the  likelihood  of 
reunification.  Of  the  45  agencies  represented  in  the  family-based  model,  34  have  results  that 
are  significantly  above  or  below  the  comparison  agency  in  the  base  case.  However,  the 
addition  of  community  needs  scores  reduces  the  number  of  agencies  with  significant 
reunification  effects  to  20.  Among  congregate  care  agencies,  the  introduction  of  community 
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needs  data  has  moderate  effects  on  the  number  of  agencies  with  significantly  greater  or  less 
likelihood  of  reunification. 

Table  4  presents  data  from  the  re-entry  analyses,  for  both  family-based  and  congregate 
care  models.  Results  in  the  base  model  largely  fall  in  directly  opposite  patterns  as  those  in  the 
Reunification  base  model — older  children,  those  not  Federally  eligible,  African-American 
children,  and  those  experiencing  prior  stays  in  care  greater  than  60  days  all  have  diminished 
likelihood  of  Re-entering  care.  These  patterns  largely  hold  for  both  Family-  and  Congregate- 
Care  samples. 

The  addition  of  community  needs  scores  to  the  basic  models  has  few  effects  on  the 
demographic  and  placement  experience  independent  variables  in  the  re-entry  models,  as  was 
the  case  for  the  reunification  models.  Overall,  adding  community  needs  to  the  family-based 
care  model  appears  to  reduce  the  effect  of  age  on  the  model’s  abihty  to  predict  re-entry,  and 
the  Hazard  Ratio  for  Kinship  Care  drops  from  0.93  to  0.49,  indicating  that  children  from 
Kinship  foster  homes  have  much  less  likelihood  of  re-entering  care  once  discharged  (p<.01). 
Introducing  community  needs  into  the  congregate  care  analysis  had  few  effects. 

When  community  needs  scores  are  entered  as  a  continuous  variable,  community  needs 
are  not  significant  in  the  family-care  model,  but  are  significant  in  the  congregate  care  sample 
(p<.01).  In  both  cases  likelihood  of  re-entry  rises  as  community  need  increases.  Quartile 
analysis  of  community  needs  were  run  in  separate  models,  but  only  the  coefficients  relating  to 
these  variables  are  reported  here,  as  other  control  variables  generated  similar  Hazard  Ratios 
and  patterns  of  significance.  This  analysis  finds  hazard  ratios  greater  than  1 .00  for  all  three  of 
the  least  needy  quartile  groupings. 
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Table  3.  Results  from  Proportional  Hazards  Analysis,  using  Cox  Regression — Reunification  Models. 

Reunification-Family  Care  Reunification-Congregate  Care 


Number  of  Observations  1 5 ,606 

Overall  Significance  (Probabilitv>Chi2)  0.0000 

Hazard 

Ratio 

Gender  ("Base:  Malel 

Female  1 .07 

Age  tBase:  Age  <n 

Age  1  1.23 

Age  2-4  1.32 

Age  .5-7  1.33 

Age  8-10  1.41 

Age  11-13  1..33 

Age  14-17  1..33 

Age  18+  1.94 

Level  of  Care  ('Base:  Foster 

Kinship  Foster  Care  0.62 

Group  Residence  ( 1 3-25  Children") 

Group  Homes  (^7-1 2  Children") 

Agency  Homes  0-6  Children") 
Residential  Treatment 
Financial  ligibilitv  TBase:  Federal  TV-  ") 

ligibilitv-Not  Federal  2.60 

ligibilitv-  mergencv  Assistance  3.05 

1  i  gibi  1  i  t  v-Other  5 . 26 

ligibilitv-Missing  0.00 

Admission  Year  ('Base:  1995") 

Admission  Year  1996  0.91 

Admission  Year  1997  0.87 

Race  tBase:  White") 

Race-Black  0.79 

Race-Hi  spani  c  0. 89 

Race-Other  0.90 

Race-Unknown  0.83 

Siblings  ('Base:  no  siblings") 

Sihlings-Not  in  Care  1 .20 

Siblings-Tn  Care  0.83 

Replacements  in  Care  ('Base:  No 
Child  xperiences  Replacement's)  0.79 

Agency  ffects: 

Number  Agencies-Not  Significant  1 1 

Number  Agencies.  Significance<.  1 0  34 

Significance  Level  ( z)<.  10  3 

Significance  Level  (z)<.05  1 1 

Significance  Level  (z)<.01  5 

Significance  Level  (z)<.001  15 

Community  Need-Continuous  Scale 

Community  Need-Ouartiles 

First  Ouartile 
Second  Ouartile 
Third  Ouartile 


11,074 

3,829 

2,022 

0.0000 

0.0000 

0.0000 

Hazard 

Hazard 

Hazard 

Ratio 

Ratio 

Ratio 

1.01 

1.051 

1.02 

1.22 

**** 

1.606 

3.40 

* 

1.3.3 

**** 

1.745 

3.07 

** 

**** 

1.34 

**** 

1.770 

2.84 

** 

1.49 

**** 

1 .397 

1..54 

1.29 

**** 

2.011 

** 

3.47 

*** 

1.26 

*** 

2.463 

*** 

4.71 

**** 

2.07 

*** 

7.227 

** 

0.64 

**** 

1.004 

0.75 

1.264 

*** 

1.19 

0.905 

0.98 

0.068 

*** 

2.4.5 

**** 

0.990 

1.26 

3.28 

**** 

1 .553 

**** 

1.62 

**** 

4.41 

**** 

6.145 

**** 

2.06 

0.00 

0.257 

0.97 

0.988 

1.88 

**** 

0.96 

0.929 

1.91 

**** 

0.82 

*** 

0.687 

* 

0.81 

** 

0.91 

0.89.3 

1.08 

1.01 

0.925 

0.44 

*** 

0.87 

* 

0.790 

1.09 

1..37 

**** 

0.934 

0.80 

**** 

0.90 

*** 

0..571 

**** 

0.92 

0.73 

**** 

0.478 

**** 

0.97 

25 

25 

26 

20 

11 

7 

4 

4 

2 

2 

5 

2 

8 

2 

1 

6 

-- 

2 

Q  97*** 

1.04 

1.18 

1.10* 

1.02 

1  1 9  ^  ^ 

0.95 

Community  Need  in  Outcomes  Evaluations 


82 


Lastly,  introducing  community  needs  scores  into  the  re-entry  models  appears  to  have 
some  effect  on  the  number  of  contract  family-based  foster  care  agencies  with  re-entry  rates 
significantly  higher  or  lower  than  the  median  agency.  Of  41  agencies  represented  in  the 
family-based  model,  21  were  significantly  above  or  below  the  comparison  agency  in  the  base 
case.  However,  the  addition  of  community  needs  scores  reduces  the  number  of  agencies  with 
significant  reunification  effects  to  15.  Among  congregate  care  agencies,  introducing 
community  needs  data  has  modest  effects  on  the  number  of  agencies  found  to  have 
significantly  greater  or  less  likelihood  of  re-entry. 

Discussion 

This  analysis  has  several  important  limitations,  requiring  cautious  interpretation  of  any 
findings.  The  first  limitation  which  should  be  noted  is  the  substantial  number  of  cases  with 
missing  address  data,  a  pattern  which  t-tests  reveal  as  associated  with  significant  bias  to 
independent  and  dependent  variables  used  in  these  models.  While  the  very  large  sample  sizes 
and  use  of  propensity  weights  in  the  survival  analysis  models  ameliorates  the  impact  of 
missing  data,  the  analysis  presented  here  should  probably  be  considered  cautiously  until 
verified  by  more  complete  data. 
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Table  4.  Results  from  Proportional  Hazards  Analysis,  using  Cox  Regression — ^Re-Entry  Models. 


Re-Entrv 

Familv  Care 

Re-Entrv  Conj 

tregate  Care 

Number  of  Observations 

13,284 

8,950 

2,163 

1,382 

Overall  Significance  (Probability>Chi2) 

0.0000 

0.0000 

0.0000 

0.0000 

Hazard 

Hazard 

Hazard 

Hazard 

Ratio 

Ratio 

Ratio 

Ratio 

Gender  (Base:  Male"! 

Female 

1.01 

0.89 

1 

1.21 

A?e  fBase:  Age  <n 

Age  1 

1.16 

1.18 

1.03 

3.81  ** 

Age  2-4 

0.81* 

0.00**** 

2.0.3 

Age  5-7 

0.57**** 

064**** 

0.56 

2..39* 

Age  8-10 

0  71 *** 

0.74** 

2.04 

1.85 

Age  11-13 

0^91 

0.97 

2.31 

1.22 

Age  14-17 

0.54*** 

0.88 

0.73 

1.44 

Level  of  Care  ^Base:  Foster  Home/Institution) 

Kinshin  Foster  Care 

0.93 

0  49*** 

Groun  Residence  G  3-25  Children") 

1.02 

1.16 

Groun  Homes  ("7-1 2  Children^ 

2.51  *** 

1.08 

Agencv  Homes  G-6  Children") 

1..59 

1.29 

Financial  ligibilitv  fBase:  Federal  TV-  ") 

ligibilitv-Not  Federal 

Q  21  **** 

Q  21  **** 

0..5.5** 

0.51 

ligibilitv-  mergencv  Assistance 

0  25  **** 
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More  broadly,  interpretation  of  the  results  of  this  analysis  is  also  limited  by  a  lack  of 
case  mix  or  risk  adjustments  in  the  data  available  in  these  settings,  which  makes  more 
complex  the  interpretation  of  results  regarding  the  relationship  between  community  need  and 
foster  care  outcomes  presented  here.  Specifically,  these  models  lack  appropriate  adjustments 
for  a  variety  of  child  and  family  risk  factors  which  are  widely  known  to  affect  service 
outcomes  in  these  settings,  including  such  domains  as  substance  abuse,  health  and  mental 
health  needs,  domestic  violence,  or  housing  needs.  In  a  sense,  the  failure  to  include  these 
variables  which  the  research  literature  would  widely  associate  with  the  outcomes  of  interest, 
could  represent  a  substantial  specification  error  in  these  models.  Such  errors  may  introduce 
substantial  bias  into  hazard  ratios  reported  here 

In  effect,  community  need  scores  are  serving  “double  duty”  in  the  event  history 
models  presented  here.  On  one  hand,  community  need  scores  serve  as  a  proxy  or  instrumental 
variable,  representing  the  range  of  unmeasured  risk  adjustments  which  should  be  included  in  a 
fully  specified  model.  While  the  extent  of  missing  values  for  this  variable  and  the  challenges 
of  developing  valid  measures  of  community  need  may  somewhat  affect  the  precision  of  these 
measures,  the  use  of  weighting  methods  to  adjust  for  missing  data  and  the  broad  support  for 
the  community  need  scales  within  the  subject  jurisdiction  should  increase  confidence  in  the 
precision  of  these  measures.  Most  importantly,  including  community  need  should  represent  a 
clear  and  substantial  improvement  in  the  specification  of  these  outcomes  models,  and  reduce 
bias  in  hazard  ratios  calculated  in  the  event  history  analyses  presented  here.  This  impact  may 
be  most  clearly  reflected  in  child  race/ethnicity  and  service  agency  dummy  variables  which 
both  have  reduced  patterns  of  significance  when  community  need  scores  are  introduced. 
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Finally,  the  scope  of  outcome  measures  used  as  dependent  variables  may  also 
represent  a  limitation  of  this  work.  One  set  of  limitations  in  this  area  are  the  measures  used 
here;  for  example,  quicker  Reunification  of  children  in  foster  care  with  their  families  is  a 
beneficial  outcome,  only  to  the  extent  that  the  child  is  safe  and  well  cared  for  at  all  times.  At 
the  same  time,  even  if  safety  and  well-being  of  the  child  can  be  assured  upon  return  to  their 
family,  it  is  possible  that  the  resources  and  support  available  from  foster  caregivers  would 
provide  additional  benefit  to  the  child’s  development  relative  to  what  is  that  child  is  likely  to 
experience  in  their  family.  More  broadly,  the  outcome  measures  reflected  in  this  model  fail  to 
account  for  other  observable  outcomes  which  may  be  of  interest  (e.g.,  child  transfers  within 
placement,  or  unexcused  absences  from  care)  or  for  other  outcomes  which  might  be  harder  to 
objectively  observe  but  which  could  be  subjectively  assessed,  such  as  child  behaviors, 
parenting  activities/skills,  health/mental  health  status,  or  substance  abuse  within  the  family. 

At  the  same  time,  statistically  significant  findings  are  demonstrated  that  community 
need  is  associated  with  some  child  foster  care  outcomes.  As  a  result  of  this  dual  role,  it  is  not 
possible  at  this  time  to  determine  whether  community  needs  themselves  are  impacting  on 
child  foster  care  outcomes  and/or  whether  their  impact  is  derived  from  their  service  as  proxies 
for  unmeasured  child  and  family  needs,  risks  and  strengths. 

Even  with  these  limitations,  this  analysis  offers  valuable  insights  on  several  domains. 
At  the  broadest  level,  this  research  offers  limited  but  measurable  support  for  the  impact  of 
community  needs  on  children's  likelihood  of  reunification  from  family-based  foster  care,  the 
most  common  service  model  used  for  children  in  out-of-home  care.  This  finding  lends 
support  to  policy  analysts,  managers  and  advocates  who  have  called  for  the  adoption  of 
community-based  child  and  family  service  models.  The  lack  of  significant  associations 
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between  community  needs  and  child  Reunification  in  congregate  care  settings  may  reflect  the 
greater  clinical  service  needs  of  children  placed  in  these  more  intensive  settings.  Higher 
community  need  scores  are  also  associated  with  increased  risk  of  Re-entry  in  Congregate  Care 
settings. 

Incorporating  community  needs  also  appears  to  offer  benefit  in  evaluating  foster  care 
services.  One  evocative  finding  in  this  vein  is  the  reduction  in  the  impact  of  race  on  outcomes 
in  several  of  these  models.  While  this  effect  is  not  definitive  and  may  in  part  reflect  the 
substantial  interaction  between  race  and  community  needs  across  the  range  of  New  York 
City’s  59  community  districts,  it  offers  limited  support  for  incorporating  community  need 
characteristics  into  models  which  include  ethnic  or  racial  categories  as  control  variables,  and 
may  explain  some  of  the  apparent  association  between  ethnic/racial  group  membership  and 
foster  care  outcomes  [Courtney,  Barth  and  Park,  1996]. 

Also  notable  is  the  impact  community  need  has  on  agency  effects  in  these  models. 
While  the  addition  of  community  needs  had  little  effect  on  observed  outcomes  for  children  in 
congregate  care,  there  were  shifts  at  the  family-care  level  in  both  reunification  and  re-entry 
models.  The  number  of  service  agencies  with  reunification  outcomes  significantly  above  or 
below  average  dropped  from  34  to  20  when  community  needs  were  added  to  the  model.  The 
re-entry  model  saw  a  reduction  from  21  to  15  agencies  with  significant  results.  This  would 
appear  to  imply  that  differences  in  agency  service  quality  or  programmatic  approach  has  less 
impact  on  child  outcomes,  once  community  context  is  considered. 

The  results  of  this  analysis  also  highlight  the  need  to  incorporate  risk  adjustments 
when  evaluating  outcomes  in  child  and  family  service  programs,  and  the  potential  usefulness 
of  proxy  measures  when  policy  researchers  are  working  in  real-world  settings  where  not  all 


Community  Need  in  Outcomes  Evaluations 


87 


needed  data  may  be  available.  In  this  sense,  the  community  needs  data  incorporated  in  this 
analysis  are  used  as  a  proxy  for  clinical  or  program  service  need  data  on  children  and/or 
family  members. 

Future  work  in  this  area  could  focus  in  two  directions.  First,  replicating  this  analysis 
with  more  complete  address  and/or  community  needs  data  might  yield  more  reliable 
assessments  of  the  relationship  between  service  outcomes  and  community  needs.  Future 
analysis  should  also  seek  to  incorporate  measures  of  child  and  family  service  needs  in 
addition  to  community  need  scores.  This  would  allow  evaluation  models  to  account  for  a 
wider  range  of  clinical  and  other  service  needs  of  children  and  families  served  by  these 
programs,  would  allow  more  precise  examination  of  whether  community  needs  scores  can 
serve  as  proxies  for  more  complex  clinical  needs  data,  and  penmt  more  complete  conclusions 
to  be  drawn  as  to  the  relationship  between  community  needs  and  service  outcomes  in  foster 
care. 
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Chapter  7.  Evaluating  Foster  Care  Services:  Integrating  Qualitative  and  Quantitative 
Methods 

Introduction 

This  paper  seeks  to  contribute  to  our  understanding  of  evaluation  methods  in  foster 
care  services  by  examining  the  impact  of  quality-of-care  measures  on  child  foster  care 
outcomes,  using  data  from  three  analytic  tools  developed  by  the  New  York  City 
Administration  for  Children’s  Services  (ACS).  Among  the  many  challenges  facing  those 
interested  in  impacting  U.S.  foster  care  systems  is  developing  valid,  reliable  methodologies 
for  evaluating  the  Quality  of  outcomes  for  the  children,  families  and  communities  affected  by 
these  services.  One  set  of  approaches  to  foster  care  program  evaluation  have  focused  on  the 
use  of  administrative  data  systems  to  assess  child  outcomes  from  foster  care,  typically  using 
event  history  proportional  hazards  methodologies  [Wulczyn,  Orlebeke,  and  Melamid,  2000; 
Fraser,  Pecora,  and  Popuang,  1992].  Other  researchers  have  emphasized  the  need  to  include  a 
broader  range  of  measures  in  any  effective  foster  care  program  evaluation. 

One  very  comprehensive  framework  suggested  by  Courtney  [1993]  highlights  the 
importance  of  foster  care  evaluations  including  measures  of  program  structural  characteristics 
(including  physical  plant,  staff  ratios,  and  staff  qualifications/training),  program  process 
characteristics  (including  timeliness  of  assessment  and  treatment  planning,  and  quality  and 
timeliness  of  all  casework  contacts  and  visitation),  and  child  and  family  outcomes  (including 
reunification,  reentry,  adoption,  and  transfer/absence  from  care  events,  as  well  as  client 
behavioral,  emotional,  and  attitudinal  shifts  and  client  satisfaction  measures).  However,  this 
author  acknowledges  that  such  comprehensive  reviews  are  particularly  challenging  given:  (a) 
the  lack  of  child  and  family  need  assessment  tools  with  test-retest  validity,  (b)  the  complexity 
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and  cost  of  comprehensive  evaluation,  and  (c)  the  diversity  and  lack  of  standardization  in  case 
reporting  practices  [Courtney,  1993,  p.28;  Magura  and  Moses,  1986]. 

There  are  a  number  of  reasons  for  the  relative  absence  of  child  and  family  well  being 
measures  in  these  organizations,  including  the  difficulty  of  defining  and  measuring  clinical 
and  service  needs,  reluctance  on  the  part  of  researchers  and  program  managers  to  use  tools 
which  might  negatively  label  children  and  families  and/or  set  unrealistic  expectations  for 
systems  to  “fix”  children  and  families  with  severe,  complex  and  the  long-standing  problems 
[Altshuler  and  Gleeson,  1999,  p.l28].  Development  of  client  need  assessment  tools  and 
implementation  of  comprehensive  program  evaluation  systems  in  child  welfare  requires 
researchers  and  program  managers  to  balance  organizational  culture  and  evaluation 
methodologies  [Usher,  1995].  Ultimately,  development  of  comprehensive  need  assessment 
and  risk  management  systems  may  well  represent  the  most  substantial  challenge  facing  child 
welfare  systems  today  [Gambrill  and  Shlonsky,  2001]. 

This  study  reports  correlation  scores  for  each  measure  at  the  program  level, 
summarizes  reunification,  re-entry  and  adoption  finalization  rates,  and  also  uses  proportional 
hazards  analysis  of  data  available  on  individual  children  within  this  system  to  examine  the 
effect  certain  of  these  Quality  of  Care  measures  have  on  outcomes  for  children  in  foster  care. 
A  goal  of  this  work  is  to  identify  strategies  available  to  managers,  evaluators,  advocates  and 
clients  who  wish  to  better  understand  the  effectiveness  of  different  programs,  as  well  as 
suggesting  useful  strategies  for  child  service  systems  that  are  just  beginning  to  develop 
enhanced  program  evaluation  tools  for  foster  care.  Finally,  this  study  may  offer  insights  into 
other  approaches  that  researchers  and  evaluators  may  wish  to  explore  to  continue  improving 
available  techniques  for  evaluating  foster  care  services. 


Community  Need  in  Outcomes  Evaluations 


90 


Methodology 


Setting:  As  noted  above,  data  used  in  this  analysis  are  from  the  New  York  City 
Administration  for  Children’s  Services  (ACS),  the  nation’s  second  largest  foster  care 
jurisdiction  [Needell  et  al.,  2001].  Beyond  its  sheer  size,  this  service  jurisdiction  is  relatively 
unique  in  its  reliance  on  more  than  50  separate  contract  non-profit  agencies  to  provide  out-of- 
home  placement  services  for  approximately  three-fourths  of  children  in  its  care  [ACS 
Monthly  Indicator  Report,  April  2001;  Melamid  et  al.,  1998].  While  few  jurisdictions  have 
this  substantial  a  reliance  on  contract  service  agencies,  this  setting  offers  an  excellent 
opportunity  to  examine  effects  of  program  Quality  on  service  outcomes  across  a  large  number 
of  separate  service  programs  operating  within  one  jurisdiction. 

Data  Sources:  ACS’  foster  care  program  evaluation  structure  relies  on 
triangulation — the  use  of  multiple,  well-designed  measures  to  enhance  and  confirm  the 
validity  of  evaluation  scores  for  any  subject  [Campbell  and  Fiske,  1959;  Crano,  1981;  Greene 
and  McClintock,  1985;  McClintock  and  Greene,  1985].  The  three  sources  of  information 
used  in  this  analysis  are:  (1)  Compliance/Timeliness  of  administrative  activities  of  contract 
agencies  (Contractor  Overall  Performance  Evaluation  System),  (2)  quantitative  outcomes 
measures  (ACS/Chapin  Hall  Outcomes  Analysis),  and  (3)  Quality  of  Care  reviews  (Program 
Evaluation  System).  Using  multiple  approaches  to  combine  evaluation  results  for  the  same 
subjects  not  only  provides  more  accurate  overall  evaluation  results,  but  can  also  provide  more 
detailed  portraits  of  the  strengths  and  weaknesses  of  programs  being  evaluated.  Figure  1 
sununarizes  the  basic  characteristics  of  these  three  methodologies. 
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Figure  1.  Foster  Care  Program  Evaluation  Methodologies — New  York  City  Administration 
for  Children’s  Services. 
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Compliance/Timeliness  Measures:  ACS’  longest-established  method  for  assessing 
the  quality  of  services  delivered  by  contract  and  directly  managed  foster  care  agencies  is  its 
Contractor  Overall  Performance  Evaluation  System,  which  generates  agency  administrative 
Compliance/Timeliness  scores.  As  can  be  seen  in  Figure  1,  the  Compliance/Timeliness 
measures  fall  into  three  broad  categories — ^timeliness  of  completion  of  casework  activities, 
success  or  failure  in  achieving  key  service  objectives,  and  assessments  of  contract  agency 
fiscal  management. 

Within  each  of  these  categories,  data  are  collected  from  administrative  reporting 
systems  operating  in  a  variety  of  agency  program  development  and  oversight  units.  Each 
service  agency  is  measured  on  all  indicators  that  are  appropriate  to  their  scope  of  service,  and 
overall  agency  Compliance/Timeliness  scores  are  tabulated  for  each  agency  calculated  on  a 
scale  from  0.00  to  1.00.  Data  reported  in  this  study  were  collected  in  1998  and  reported  in 
1999,  so  as  to  match  data  collected  using  the  other  methodologies. 

Quality  of  Care  Measures:  One  effort  to  enhance  ACS  program  evaluation  capacity 
surrounded  the  creation  of  validated  tools  to  review  Quality  of  program  services  for 
statistically  valid  samples  of  case  records  in  each  foster  care  agency.  The  Program 
Evaluation  System  (PES)  uses  multiple  instruments  to  gather  information  from  different 
sources  to  discern  “meaningful  differences  in  Program  Quality”  across  foster  care  programs 
[Wells,  1999].  Development  of  these  measures  grew  out  of  an  expert  panel  which  identified  a 
list  of  approximately  700  variables  that  could  be  included  in  a  foster  care  review  system. 
Researchers  then  developed  a  comprehensive  set  of  case  review,  site  observation  and 
structured  interview  questionnaires  designed  to  collect  data  on  these  variables,  and  a  designed 
sampling  plan.  Two  pilot  studies  were  conducted  to  collect  these  data,  which  were  then 
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subject  to  analysis  of  consistency  and  reliability,  as  well  as  qualitative  item  review,  using  the 
judgment  of  consulting  staff  who  developed  the  Quality  scales  [Wells,  1999,  p.4-1].  This 
pilot  study  resulted  in  a  reduced  set  of  instruments  and  items  used  in  the  final  PES,  reported 
for  the  first  time  by  ACS  in  1998,  based  on  case  reviews  conducted  in  1996-1997. 

The  Quality  of  Care  tools  collect  data  from  a  random  sample  of  cases  representing 
children  in-care  with  a  given  level  of  care  or  program  type  within  each  direct  care  or  contract 
service  agency  (e.g.,  traditional  foster  boarding  homes,  kinship  foster  boarding  homes,  group 
homes,  agency  boarding  homes,  institutions).  For  each  child  selected  for  review,  a  dedicated 
team  of  ACS  staff  completes  case  record  reviews,  site  observations  and  structured  interviews 
with  the  social  workers,  child  care  workers,  foster  parents  and  parents.  Program  and  agency 
scores  are  derived  by  aggregating  scores  across  each  child  within  each  analysis  unit. 

Quality  of  Care  data  represent  three  conceptual  subscales: 

•  Basic  Welfare,  reflecting  the  programs  ability  to  ensure  physical  safety  and  basic 
needs  of  the  child; 

•  Normal  Growth  and  Development,  reflecting  care  that  children  require  to  become 
functioning  members  of  society,  such  as  schooling,  recreation  and  nurturing 
relationships;  and 

•  Process,  reflecting  responsibilities  of  program  staff,  foster  caregivers  and  other 
program  actors  that  are  linked  to  positive  outcomes  for  children  in  care,  including 
service  planning,  provision  of  needed  supports  and  services,  client  contacts,  and 
family  visitation  [Wells,  1999]. 
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A  fourth  sub-scale,  Client  Satisfaction,  was  dropped  from  these  analyses  due  to  lack  of 
consistent  and  reliable  measurement  qualities.  ACS  and  contract  agency  staff  found  results  at 
the  subscale  level  difficult  to  interpret,  and  so  have  developed  revised  scales  from  specific 
items  contained  within  the  case  record  review  data,  including  scales  for  reunification  services, 
service  planning,  casework  contacts  and  visitation,  and  assessments  [Edell,  2001], 

Quantitative  Outcome  Indicators:  The  final  element  in  the  ACS  program  evaluation 
strategy  was  the  implementation  of  quantitative  measures  of  child  outcomes  from  the  foster 
care  system.  This  approach,  which  was  jointly  developed  by  ACS  and  the  Chapin  Hall  Center 
for  Children  at  the  University  of  Chicago,  relies  on  ACS’  administrative  database  to  evaluate 
several  key  experiences  children  have  in  moving  through  the  foster  care  system  [Wulczjm, 
Orlebeke,  and  Melamid,  2000].  This  analysis  uses  event  history  analysis  methods  to  evaluate: 

•  reunification  rates,  measured  by  the  likelihood  of  children  returning  to  their 
parents  or  primary  resource  person  given  the  amount  of  time  they  have  been  in 
care; 

•  re-entry  rates,  the  risk  of  a  child  returning  to  foster  care  within  one  year  of  a 
discharge  home;  and 

•  adoption  finalization  rates,  reflecting  the  likelihood  of  a  child  with  a  goal  of 
adoption  experiencing  their  final  adoption  in  any  given  time  period. 

The  reunification  and  adoption  models  use  cohorts  of  children  admitted  to  care  over  a 
given  period,  while  the  re-entry  model  focuses  on  cohorts  of  children  discharged  from  care. 
All  models  control  for  a  variety  of  child  demographic  and  placement  history  characteristics 
recorded  in  the  ACS  database  [Wulczyn,  Orlebeke  and  Melamid,  2000;  Fraser,  Pecora,  and 
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Popuang,  1992].  The  analysis  of  the  Reunification  rates  reported  here  focus  on  children  first 
admitted  to  care  in  1995, 1996,  and  1997,  while  the  analysis  of  Re-entry  rates  focuses  on  the 
cohort  of  children  experiencing  their  first  returns  home  in  1996, 1997  and  1998.  The 
Adoption  analysis  uses  cohorts  of  children  first  admitted  in  1992, 1993  and  1994  to  better 
reflect  the  length  of  time  required  for  a  child  to  move  into  and  through  the  adoption  process. 

Analytic  Methods:  The  first  set  of  analyses  presented  in  this  paper  are  descriptive 
statistics  and  correlations  of  calculated  scores,  across  the  three  methods  described  above,  for 
each  agency.  These  analyses  provide  basic  measures  of  the  results  from  the  different  models, 
and  will  be  used  to  highlight  commonalities  and  differences  across  programs  assessed  using 
these  tools.  Measures  of  central  tendency  are  also  presented  for  reunification,  re-entry  and 
adoption  finalization  rates  and  length  of  time  to  outcomes  across  quartiles  of  agency 
performance  on  both  the  CompUance/Timeliness  and  Quality  of  Care  measures. 

The  remaining  analyses  reported  here  explicitly  build  upon  the  proportional  hazards 
Outcomes  models  developed  by  ACS  and  Chapin  Hall  and  described  in  the  previous  chapter 
of  this  work  [Wulczyn,  Orlebeke  and  Melamid,  2000].  This  work  supplements  these  basic 
Outcomes  models,  adding  agency-level  performance  scores  from  the  Compliance/Timeliness 
and  Quality  of  Care  measures  to  assess  whether  program  level  measures  of  service  quality 
contribute  significantly  to  reunification,  re-entry  and  adoption  outcomes  for  children.  All 
basic  models  are  also  supplemented  by  the  inclusion  of  community  need  scores,  reflecting  the 
impacts  measured  community  level  risks  may  have  on  foster  care  outcomes.  As  reported  in 
Chapter  Six  of  this  work,  these  community  need  measures  have  been  found  to  have 
statistically  significant  effects  on  some  results  obtained  from  these  models,  hi  this  model, 
these  measures  also  serve  as  proxies  for  child  and  family  service  need  risk  adjustments  which. 
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though  difficult  to  reliably  measure,  should  be  considered  in  evaluating  outcomes  from 
services  to  at-risk  populations  [Courtney,  1993;  Altshuler  and  Gleeson,  1999]. 

Estimating  these  models  at  the  individual  child  level  was  considered,  but  was  not 
feasible,  primarily  due  to  samples  that  were  too  small  and  too  systematically  biased  when 
Quality  and  Outcome  data  were  merged.  In  addition,  Compliance/Timeliness  were  only 
available  at  the  program  level,  but  not  at  the  individual  child  level.  In  this  case  bias  arises 
from  the  different  sampling  frames  used  for  the  datasets  in  this  analysis:  while  the  Outcome 
models  are  estimated  using  data  from  annual  cohorts  of  children  experiencing  first  placements 
or  first  re-entrances.  Quality  data  are  collected  from  an  in-care  sample.  In  most  cases, 
sampling  within  in-care  populations  introduces  risk  of  skewing  placement  duration  longer  and 
reunification  rates  lower  than  would  be  found  over  a  sample  drawn  from  entry  cohorts  of 
children  [Wulczyn,  Bruner,  and  Goerge,  1999].  In  this  analysis,  merging  the  in-care,  point-in¬ 
time  Quality  of  Care  data  with  the  entry-cohort  outcomes  data  resulted  in  children  having 
longer  stays  in  care  being  over  represented  relative  to  those  with  shorter  stays,  and  an 
insufficient  number  of  cases  to  reliably  weight  data  to  correct  this  bias. 

Results 

Correlation  Analysis:  Scores  on  the  Compliance/Timeliness,  Quality  and  Outcomes 
measures  where  tabulated  for  most  of  the  61  service  agencies  under  contract  with  ACS  in 
1996-1997.  Table  1  presents  descriptive  statistics  for  the  indicators  reported  from  the  three 
evaluation  tools  described  above.  Note  that  only  data  relating  to  non-profit  contract  providers 
is  presented  in  this  analysis,  since  outcomes  data  were  not  available  for  direct  service  agencies 
during  this  time  period.  Note  also  that  the  number  of  agencies  included  in  each  of  these 
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measures  varies,  since  not  all  programs  provide  service  in  all  measured  areas,  or  due  to 
sample  sizes  too  small  to  generate  reliable  performance  measurements. 


Table  1 .  Descriptive  Statistics  for  ACS  Evaluation  Measures:  Compliance/Timeliness 
(COPES),  Quality  (PES),  and  Outcomes  (ACS/Chapin  Hall  Measures)  Scores,  1998  and  1999 

Number 

Agencies  Mean  Std.  Dev.  Min  Max 

1998  Results  (All  Services): 

Process 

59 

3.07 

0.78 

1 

4 

Outcomes:  Average 

61 

1.09 

11.48 

-30 

27 

Outcomes:  Reunification 

61 

0.41 

12.40 

-35 

43 

Outcomes:  Re-Entry 

42 

-0.45 

5.59 

-16 

13 

Outcomes:  Adoption 

47 

4.43 

26.59 

-53 

54 

Quality:  Average 

45 

0.78 

0.06 

0.59 

0.89 

Quality:  Basic  Welfare 

45 

0.85 

0.07 

0.6 

0.94 

Quality:  Normal  Growth 

42 

0.74 

0.07 

0.5 

0.88 

Quality:  Process 

42 

0.78 

0.08 

0.48 

0.87 

1999  Results 

(Family  Care  only): 

Process 

41 

0.77 

0.12 

0.49 

1.00 

Outcomes 

41 

0.65 

0.11 

0.33 

0.87 

Quality 

41 

0.81 

0.04 

0.73 

0.87 

As  can  be  seen  in  this 

table,  the  three  tools  result  in  scores 

that  are  very 

differently 

scaled.  For  example,  while  Compliance/Timehness 

scores  range  1 

from  0.49  to 

1.00,  Adoption 

Outcome  scores  range  from  - 

•53  to  +54, 

and  Quality 

of  Care  values  from  .48  to 

1 .94  across  the 

three  measures^  Quality  of  Care  results  were  much  higher  on  the  Quality/Basic  Welfare  sub- 


^  These  relative  performance  calculations  were  made  in  the  base  model  by  selecting  1 8  repeated  random  samples 
of  children  across  all  agencies,  and  assessing  the  performance  of  each  agency  relative  to  this  randomly  selected 
sample.  Agencies  were  given  a  score  from  -3  to  +3  for  each  of  these  18  iterations,  representing  the  number  of 
standard  deviations  separating  their  calculated  hazard  ratio  from  the  value  of  the  random  sample.  As  a  result 
agency  outcome  scores  can  range  from  -54  to  +54,  across  these  1 8  iterations.  The  number  of  iterations  was 
established  empirically — analysts  repeated  iterations  until  five  repeated  samples  revealed  no  shifting  in  the  rank 
ordering  of  agency  performance  on  the  most  widely  ranging  model,  reunification  from  family  care.  ACS  and 
Chapin  Hall  have  subsequently  replaced  this  multiple  iteration  approach  with  an  approach  that  uses  the  median 
performing  agency  as  the  comparison  agency  for  purposes  of  determining  relative  performance.  This  later 
approach  is  used  in  the  analyses  presented  in  this  dissertation. 
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scale  (mean=.85)  than  on  Quality/Growth  and  Development  (mean=.74)  and  Quality/Process 
sub-scales  (mean=.78).  This  result  was  expected,  given  the  substantial  regulatory  oversight  of 
agencies  in  the  areas  covered  by  the  Basic  Welfare  measure.  Also  notable  were  the 
differences  in  measures  of  dispersion  among  Outcome  measures — standard  deviations  and 
ranges  of  scores  for  Re-Entry  are  much  narrower  than  those  for  Reunification  and  Adoption, 
reflecting  less  variation  in  this  measure. 

Prior  to  performing  correlation  analysis  on  these  results,  the  normality  assumption  was 
tested  for  all  variables,  using  Beta-One  statistics  calculated  by  dividing  the  third  moment 
measure  for  this  distribution  (skewness)  by  its  second  moment  measure  (standard  error). 

Scores  between  -2  and  +2  were  considered  to  reflect  a  normal  underlying  distribution  for 
these  variables.  Quality  of  Care  Basic  Welfare,  Normal  Growth  and  Process  measures  had 
Beta-One  values  below  -2,  reflecting  their  non-normal  distribution,  strongly  skewed  towards 
the  low  end  of  the  distribution  for  these  scores.  Correlation  analyses  and  significance  tests 
across  the  seven  measures  are  presented  in  Table  Two,  with  Kendall’s  Tau_B  Correlation 
calculations  used  to  calculate  correlations  involving  the  three  variables  determined  to  have 
outlier  values,  and  Pearson  correlations  for  variables  determined  to  be  normally  distributed. 
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Tables  2.  Correlations  and  Significance  Levels:  ACS  Program  Evaluation  1999 
Compliance/Timeliness ,  1998  Quality  and  1998  Outcomes  Scores,  including  subscales. 
Agency  Level. 


Outcomes: 

Process  Reunification 


Outcomes: 

0.14 

Outcomes: 

Reunification 

0.28 

Re-Entry 

Outcomes: 

0.05 

-0.31 

Outcomes: 

Re-Entry 

0.78 

0.05 

Adoption 

Outcomes: 

0.57 

0.18 

0.08 

Outcomes: 

Adoption 

0.00 

0.23 

0.65 

Total 

Outcomes: 

0.59 

N/A 

N/A 

N/A 

Total 

0.00 

Quality: 

Basic 

Quality: 

0.28 

-0.06 

0.00 

0.04 

0.01 

Welfare* 

Basic 

0.02 

0.59 

1.00 

0.73 

0.92 

Welfare^ 

Quality: 

Normal 

Quality: 

0.12 

0.13 

-0.01 

0.21 

0.23 

0.12 

Growth® 

Normal 

0.36 

0.25 

0.96 

0.07 

0.08 

0.30 

Growth* 

Quality: 

Quality: 

0.17 

0.09 

-0.05 

0.24 

0.12 

0.29 

0.40 

Process  * 

Process “ 

0.18 

0.40 

0.73 

0.03 

0.36 

0.01 

0.00 

Quality: 

0.31 

-0.02 

0.22 

-0.25 

0.13 

N/A 

N/A 

N/A 

Total 

0.02 

0.89 

0.16 

0.09 

0.47 

“-Correlation  and  Significance  levels  for  these  items  were  calculated  using 
Kendall’s  Tau-B  statistics,  due  to  non-normal  distributions. 

Compliance/Timeliness  Correlations:  The  results  of  the  correlation  analysis  present  a 
complex  picture  of  the  three  sets  of  measures  used  in  the  ACS  evaluation  system.  Overall, 
Compliance/Timeliness  scores  are  significantly  correlated  with  agency  total  scores  for  both 
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the  Quality  of  Care  and  Outcomes  (respectively,  Pearson’s  r=0.59,  p=0.00,  two  tailed; 
Pearson’s  r=0.31,  p=0.02,  two  tailed).  While  the  correlations  between  Compliance/Timeliness 
scores  and  all  three  of  the  Quality  of  Care  sub-scales  are  positive,  only  Basic  Welfare 
correlates  with  Compliance/Timeliness  at  a  statistically  valid  level  (Pearson’s  r=0.28,  p=.02, 
two  tailed).  This  is  not  surprising,  since  the  Quality/Basic  Welfare  scale  focuses  on  the  ability 
of  service  agencies  to  meet  the  fundamental  health  and  safety  needs  of  children  in  care,  and 
thus  would  be  expected  to  track  closely  with  Process  measures  which  focus  on  timely 
completion  of  regulatory  requirements  relating  to  children  in  care.  Adoption  Finalizations  are 
significantly  correlated  with  overall  Process  score,  though  this  is  expected  given  that 
Adoption  represent  approximately  one-fifth  of  the  total  points  available  in  this  scale 
(Pearson’s  r=.57,  p=.00,  two-tailed).  Correlations  between  Process  and  the  other  two 
Outcome  Measures  (Reunification  and  Re-Entry)  are  not  significant. 

Quality  Correlations:  The  only  patterns  of  significant  correlation  found  between  the 
Quality  measures  and  the  Outcome  results  are  Adoption  Finalizations  with  the 
Quality/Process  sub-scale  (Kendall’s  Tau_B  ^.24,  p=.03,  two  tailed).  Among  the  Quality 
sub-scales,  Quality/Process  is  moderately  correlated  with  the  other  two  scales,  but 
Quality/Basic  Welfare  and  Quality/Normal  Growth  and  Development  sub-scales  are  not 
significantly  correlated  with  each  other  (Pearson’s  r=.12,  p=.30,  two-tailed). 

Summary  Statistics—Outcomes  versus  Quality  and  Process  Scores:  Tables  3a  and 
3b  present  measures  of  central  tendency  for  the  outcome  measures  used  in  the  survival 
analyses,  by  quartiles  of  agency  performance  on  the  Process  and  Quality  of  Care  measures, 
respectively.  This  analysis  shows  a  pattern  in  which  agencies  having  higher 
Compliance/Timeliness  scores  tend  to  have  improved  outcomes.  Among  family-based 
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service  programs,  reunification  rates  increase  (from  70  percent  observed  reunified  to  78 
percent)  and  time  in  care  declines  (from  557  to  428  days)  across  the  four  quartiles,  while  re¬ 
entry  rates  are  notably  lower  for  children  served  by  agencies  which  receive  the  highest 
Compliance/Timeliness  scores,  though  agencies  in  quartiles  two  and  three  unexpectedly  have 
higher  re-entry  rates  than  those  in  the  lowest  quartile.  Patterns  for  congregate  care  programs 
are  similar,  though  somewhat  more  mixed.  Finally,  adoption  finalization  rates  also  generally 
increase  as  Compliance/Timeliness  performance  rises,  with  days  to  adoption  declining  by 
almost  one  year  from  the  lowest  to  the  highest  quartiles  (from  1799  to  1482  days). 

Quality  of  Care  measures  reported  in  Table  3b  reveal  a  much  less  clear  relationship  to 
outcomes.  Among  family-based  service  programs,  there  is  relatively  little  variation  in 
reunification,  reentry  or  adoption  rates  or  timeliness  across  service  Quality  quartiles.  At  the 
congregate  care  level,  generally  higher  reunification  rates  are  seen  as  Quality  of  Care 
measures  rise,  while  length  of  stay  in  care  also  rises  by  a  significant  degree  (from  178  to  304 
days).  This  later  result  is  not  necessarily  negative,  as  it  might  signify  reduced  absences  or 
transfers  for  programs  with  higher  quality  scores. 

Program  Level  Proportional  Hazards:  Analysis  of  the  impacts  of 
Compliance/Timeliness  and  Quality  measures  on  child  foster  care  outcomes  was  undertaken 
using  proportional  hazards  event  history  models,  implemented  using  Stata's  Cox  survival 
analysis  protocols  [Stata  Press,  1997].  As  noted  above,  the  basic  Outcome  models  were 
originally  developed  to  support  evaluation  of  New  York  City's  foster  care  programs 
[Wulczyn,  Orlebeke,  and  Melamid,  20(X)].  The  models  presented  here  are  amended  from  this 
basic  model  in  two  ways:  (1)  community  needs  scores  are  added  to  serve  as  a  proxy  for  child 
and  family  risk  adjustments  [Melamid,  2001];  (2)  relevant  Quality  of  care  and  agency 
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Compliance/Timeliness  measures  are  added  to  assess  the  impact  of  measured  service 
performance  on  foster  care  outcomes  for  children  in  this  sample. 
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Table  3a.  Mean  and  Standard  Deviation  Outcomes  &  Process  Scores 


Days  to  Reunified 

Family-Based  Care 

Coneresate  Care 

Obs 

Mean 

Std.  Dev. 

Obs 

Mean 

Std.  Dev. 

Process  Quartile  1 

3591 

566.57 

582.87 

840 

184.74 

325.24 

Process  Quartile  2 

3910 

508.14 

557.07 

729 

264.98 

393.44 

Process  Quartile  3 

3791 

449.61 

524.47 

794 

140.57 

248.82 

Process  Quartile  4 

2758 

428.46 

495.78 

729 

397.25 

436.96 

Process  Missing 

850 

388.88 

361.14 

21 

281.76 

406.17 

Percent  Reunified 

Process  Quartile  1 

3591 

0.70 

0.46 

0.57 

0.50 

Process  Quartile  2 

3910 

0.71 

0.45 

729 

0.59 

0.49 

Process  Quartile  3 

3791 

0.76 

0.43 

453 

0.55 

0.50 

Process  Quartile  4 

2758 

0.78 

0.42 

729 

0.60 

0.49 

Process  Missing 

850 

0.52 

0.50 

21 

0.57 

0.51 

Davs  to  Reentry 

Process  Quartile  1 

2734 

367.60 

84.19 

497 

345.28 

104.89 

Process  Quartile  2 

2956 

366.05 

84.59 

386 

349.27 

94.13 

Process  Quartile  3 

2846 

363.53 

88.54 

453 

344.95 

99.86 

Process  Quartile  4 

2038 

375.57 

69.00 

410 

343.80 

103.57 

Process  Missing 

450 

366.14 

78.12 

12 

388.00 

27.71 

Reentry  Rate 

Process  Quartile  1 

2734 

0.10 

0.31 

497 

0.14 

0.35 

Process  Quartile  2 

2956 

0.11 

0.32 

386 

0.11 

0.32 

Process  Quartile  3 

2846 

0.13 

0.34 

453 

0.17 

0.37 

Process  Quartile  4 

2038 

0.09 

0.29 

410 

0.11 

0.32 

Process  Missing 

450 

0.14 

0.35 

12 

0.00 

0.00 

Days  to  Adoption 

- 

Process  Quartile  1 

1146 

1798.83 

558.84 

Process  Quartile  2 

1310 

1689.33 

534.37 

Process  Quartile  3 

1066 

1685.31 

536.56 

Process  Quartile  4 

689 

1481.77 

535.70 

Process  Missing 

123 

1378.08 

396.74 

Adoption  Rate 

Process  Quartile  1 

1146 

0.63 

0.48 

Process  Quartile  2 

1310 

0.70 

0.46 

Process  Quartile  3 

1066 

0.70 

0.46 

Process  Quartile  4 

689 

0.78 

0.42 

Process  Missing 

123 

0.22 

0.42 
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Table  3b.  Mean  and  Standard  Deviation-  Quality  &  Outcome  Scores 


Family-Based  Care 
Obs  Mean  Std.  Obs 


Days  to  Reunified 

Quality  Quartile  1 
Quality  Quartile  2 
Quality  Quartile  3 
Quality  Quartile  4 
Quality-Missing 

Percent  Reunified 
Quality  Quartile  1 
Quality  Quartile  2 
Quality  Quartile  3 
Quality  Quartile  4 
Quality-Missing 

Days  to  Reentry 
Quality  Quartile  1 
Quality  Quartile  2 
Quality  Quartile  3 
Quality  Quartile  4 
Quality-Missing 

Reentry  Rate 
Quality  Quartile  1 
Quality  Quartile  2 
Quality  Quartile  3 
Quality  Quartile  4 
Quality-Missing 

Days  to  Adoption 
Quality  Quartile  1 
Quality  Quartile  2 
Quality  Quartile  3 
Quality  Quartile  4 
Quality-Missing 

Adoption  Rate 
Quality  Quartile  1 
Quality  Quartile  2 
Quality  Quartile  3 
Quality  Quartile  4 
Quality-Missing 


3848  494.64 

2843  501.07 

3971  459.57 

3469  507.61 

769  421.70 


3848  0.67 

2843  0.72 

3971  0.76 

3469  0.73 

769  0.77 


2755  367.63 

2086  363.08 

3101  369.47 

2437  365.41 


645 

380.53 

2755 

0.11 

2086 

0.12 

3101 

0.11 

2437 

0.12 

645 

0.08 

1624 

1691.03 

429 

1670.12 

1060 

1664.33 

993 

1685.54 

228 

1582.52 

1624 

0.63 

429 

0.75 

1060 

0.72 

993 

0.70 

228 

0.69 

542.20  953 

540.76  640 

524.56  573 

550.97  636 

504.30  311 


0.47  953 

0.45  640 

0.43  573 

0.44  636 

0.42  311 


81.37  485 

91.56  339 

78.70  372 

85.91  364 

60.71  198 


0.31  485 

0.33  339 

0.31  372 

0.33  364 

0.27  198 


552.98 

495.63 

545.75 

563.94 

571.84 


0.48 

0.43 

0.45 

0.46 

0.46 


Congregate  Care 

Mean 

Std.  Dev. 

167.87 

264.02 

195.37 

336.37 

269.87 

403.86 

304.45 

412.91 

392.92 

457.54 

0.54 

0.50 

0.58 

0.49 

0.61 

0.49 

0.61 

0.49 

0.57 

0.50 

348.20 

104.08 

339.51 

101.22 

344.15 

97.17 

352.80 

92.47 

342.84 

111.50 

0.10 

0.30 

0.20 

0.40 

0.16 

0.36 

0.11 

0.31 

0.10 

0.30 
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This  analysis  seeks  to  examine  how  agency-level  Quality  of  Care  and  Administrative 
Process  affect  children’s  outcomes  in  foster  care.  For  this  analysis  each  child  in  the  sample  is 
assigned  the  average  Compliance/Timeliness  and  Quality  scores  of  the  agency  serving  that 
child,  including  scores  from  both  the  original  subscales  and  the  alternative  subscales  within 
the  Quality  measure.  The  results  of  this  analysis  are  presented  in  Table  Four.  As  can  be  seen, 
the  five  basic  models  control  for  child  gender,  age,  race,  financial  ehgibility  (a  proxy  for 
poverty),  having  siblings  in  placement,  level  of  foster  care  placement,  admission/discharge 
year,  duration  of  time  in  care,  and  whether  the  child  experiences  multiple  placement  settings. 
The  majority  of  these  control  variables  exhibited  significant  impacts  on  child  reunification,  re¬ 
entry,  and  adoption  finalization  rates,  though  community  need  scores  were  only  statistically 
significant  for  reunification  in  family  based  settings,*  The  patterns  of  significance  and  hazard 
ratios  for  these  control  variables  closely  match  results  from  the  base  models  [Wulczyn, 
Orlebeke  and  Melamid,  2000]. 


*  Note  that  community  need  is  not  included  in  the  adoption  model  due  to  substantial  patterns  of  missing 
community  data  for  the  children  included  in  this  sample. 
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Table  4.  Proportional  Hazards  Analysis— Agency  Level  Outcomes,  Compliance/Timeliness 


and  Quality  Scores 

Reunification 


-Family  Care 

Number  of  Observations 

11,074 

Overall  Significance  (Probability>Chi2) 

0.0000 

Hazard 

Ratio 

Gender  (Base:  Male) 

Female 

1.05 

Age  (Base:  Age  <1) 

Age  1 

L27*** 

Age  2-4 

1.30**** 

Age  5-7 

\  32 

Age  8-10 

1  4Q***H< 

Age  11-13 

1.33*** 

Age  14-17 

1.29** 

Age  18 

2.10*** 

Level  of  Care  (Base:  Foster  Home/Institution) 

Kinship  Foster  Care 

0.63**** 

Group  Residence  (13-25  Children) 

Group  Homes  (7-12  Children) 

Agency  Homes  (1-6  Children) 

Residential  Treatment 

Financial  Eligibility  (Base:  Federal  IV-E) 

Eligibility-Not  Federal 

2 

Eligibility-Emergency  Assistance 

3  28 

Eligibility-Other 

4.82**** 

Eligibility-Missing 

Admission  Year  1994  (Base:  1993) 

Admission  Year  1995  (Base:  1993) 

Admission  Year  1996  (Base:  1995)  0.96 

Admission  Year  1997  (Base:  1995)  0.90* 

Discharge  Year  1998  (Base:  1997) 

Discharge  Year  1999  (Base:  1997) 

Race  (Base:  White) 

Race-Black  0.80  ** 

Race-Hispanic  0.89 

Race-Other 

Race-Unknown  0.85 

Siblings  (Base:  no  siblings) 

Siblings-Not  in  Care  1 .32  ** 

Siblings-In  Care  0.89* 

Duration  Current  Spell  in  Care  (Base-none) 

Duration  under  60  days 
Duration  60-180  days 
Duration  181-365  days 
Duration  1-2  years 
Duration  2+  years 

Duration  Last  Spell  (Base:  Less  than  31  Days) 

Duration  3 1  -60  Days 
Duration  61-180  Days 
Duration  181-365  Days 
Duration  366-730  Days 
Duration  731  or  more  Days 
Replacements  in  Care  (Base:  No  Replacements) 

Child  Experiences  Replacement(s)  0.75  *** 

Agency  Effects: 

Number  Agencies-Not  Significant  36 

Number  Agencies,  Significance<.05  9 

Community  Need-Continuous  Scale  0.97  * 
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Reunification 

Reentry  - 

Reentry  — 

Adoption  -- 

-Congregate 

Family  Care 

Congregate 

Family  Care 

2,022 

8,950 

1,382 

4,334 

0.0000 

0.0000 

0.0000 

0.0000 

Hazard 

Hazard 

Hazard 

Hazard 

Ratio 

Ratio 

Ratio 

Ratio 

0.94 

1.08 

2.65**** 

1.07 

1.30 

0.55  ***^= 

9  10**** 

0.67* 

0  43 

4.35** 

0.50** 

0.07 

0.52  ***^ 

4.36** 

0.60 

0.31 

0.58  ***^ 

5.72*** 

0.85 

0.21 

0.95 

8.01*** 

0.55 

0.06 

1.01 

33.05*** 

0.80 

1.20  * 

1.31 

1.64 

1.59*** 

1.07 

0.88 

0.13* 

0.73** 

0.30*** 

0.51** 

1.66  **** 

1.45*** 

Q  24 

0.45*** 

1.32  *** 

5.03*** 

1.05 

1.05 

0.90 

1.01 


0.90 

0.40** 

0.60*** 

0.28*** 

0.64* 

2.52* 

1.16 

0.81  ** 

0.88 

1.60 

0.46 

0.69  **= 

0.56 

2.76* 

0.79 

0.73  * 

0.79 

1.65 

0.83 

0.75  ** 

1  62**** 

0.41** 

1.24 

1.64  **^ 

0.88 

0.87 

1.13 

0.90  ** 

0.87 

0.81 

0.95 

4.37  *** 


1.15 

0.45*** 

0.50** 

0.23**** 

0.16**** 

1.22 

1.29 

0.41** 

0.21  *** 

0.49 

0.59*** 

30 

39 

31 

39 

6 

2 

5 

5 

0.95 

0.99 

0.96 

N/A 
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Process-2nd  Quartile 
Process-3rd  Quartile 
Process-Highest  Quartile 


Reunification  - 

Reunification 

Reentry  — 

Reentry  - 

Adoption  — 

-Family  Care 

-Congregate 

Family  Care 

Congregate 

Family  Care 

Hazard 

Hazard 

Hazard 

Hazard 

Hazard 

Ratio 

Ratio 

Ratio 

Ratio 

Ratio 

9  9g 

0.92 

0.93 

1  30 

3.31 

0.53  **** 

1.24 

3  JQ 

0.70 

1.54 

Number  of  Agencies  Significant<,05 
Number  of  Agencies-Not  Significant 


Quality-2nd  Quartile  Total  Score 
Quality-3rd  Quartile  Total  Score 
Quality-Highest  Quartile  Total  Score 


Number  of  Agencies  Significant<.05  27 
Number  of  Agencies-Not  Significant  1 1 


Number  of  Agencies  Significant<.05 
Number  of  Agencies-Not  Significant 


Quality — by  Subscale  Quartiles: 

Quality-  2^  Quartile  Basic  Welfare 
Quality-  3*^*^  Quartile  Basic  Welfare 
Quality-  Highest  Quartile  Basic  Welfare 
Quality-  2""*  Quartile  Normal  Growth  &  Development 
Quality-  3“^^  Quartile  Normal  Growth  &  Development 
Quality-  Highest  Quartile  Normal  Growth  &  Develop 
Quality-  2^  Quartile  Administrative  Process 
Quality-  3*^^*  Quartile  Administrative  Process 
Quality-  Highest  Quartile  Administrative  Process 

Quality — by  Alternative  Scale  Quartiles: 

Quality-  2^  Quartile  Assessments 
Quality-  3^^  Quartile  Assessments 
Quality-  Highest  Quartile  Assessments 
Quality-  2™*  Quartile  Service  Plans 
Quality-  3^“  Quartile  Service  Plans 
Quality-  Highest  Quartile  Service  Plans 
Quality-  2^  Quartile  Contacts 
Quality-  3*^^  Quartile  Contacts 
Quality-  Highest  Quartile  Contacts 
Quality-  2^  Quartile  Reunification  Services 
Quality-  3*^^  Quartile  Reunification  Services 
Quality-  Highest  Quartile  Reunification  Services 


1.24 

0.26 

0.90 

2.29 

0.06 

1.01 

0.70 

0.64 

0.90 

0.10 

1.14 

0.46 

0.28 

*** 

0.71 

0.13 

Process-2nd  Quartile 

0.77 

2.58 

1.30 

2.07 

4.47 

Process-3rd  Quartile 

2.35 

2.41 

0.95 

2.93  ** 

2.00 

Process-Highest  Quartile 

1.71 

0.36 

1.48 

3.27 

13.24 

Quality-2nd  Quartile  Total  Score 

2.10 

0.09 

1.07 

2.12  *** 

0.39 

*** 

Quality-3rd  Quartile  Total  Score 

2.25 

0.36 

*** 

0.77 

l.ll 

0.29 

Quality-Highest  Quartile  Total  Score 

1.44 

0.24 

0.57 

0.18 

Q  sH*** 

1.07 

19.28  **** 

0.046** 

1.44 

**** 

0.65  **** 

0.60 

4  38  **** 

0.768 

1.95 

**** 

0.97 

0.57 

4  52  **** 

0.003*** 

0.61 

**** 

0.90 

1.41  * 

0  29  **** 

0  007*** 

1.38 

**** 

0.75**** 

0.49  ** 

0.62  *** 

0.043** 

0.21 

**** 

0.93 

0.41  **** 

Q  32  **** 

0.525 

0.59 

**** 

1  35**** 

0  22  **** 

0  28  **** 

0.807 

0.20 

**** 

y  22*** 

1.45 

0.31  **** 

0.63 

**** 

Q 

0.67 

1.22  ** 

0.001*** 

1.65 

**** 

y  43**** 

0.50 

0.82 

1.72  **** 

0.47 

1.66 

1.18**** 

1.00 

0.72 

0.86** 

0.73 

5.92**** 

0.06 

**** 

0  ^7**** 

Q  4  J  **** 

1  54  **** 

2.10 

83.49 

**** 

0.86*** 

0.39  *** 

1 144  45  **** 

4  94  *** 

1.01 

4.29  ** 

0.32*** 

[  3  j  **** 

4.43  **** 

451  94  **** 

0.21** 

1.09 

11.86  *** 

354.48  **** 

0.34*** 

0.37 

*** 

0  61  **** 

0.64 

2.11** 

0,82**** 

0.79 

1.03 

0.85*** 

0.75 

3  95  **** 

1.44** 

47.38 

**** 
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Introducing  Compliance/Timeliness  measures  at  the  agency  level  has  generally  mixed 
results.  While  there  are  some  statistically  significant  impacts  on  child  outcomes  in  each  of  the 
five  models,  the  direction  of  these  impacts  does  not  match  expectations  in  some  cases. 
Significant  results  which  fall  in  the  expected  direction  of  relationship  included  children  served 
by  family-based  programs  in  agencies  in  the  lowest  quartile  of  Compliance/Timeliness 
performance  were  significantly  less  likely  to  experience  reunification  than  children  in  the 
other  quartiles  (p<.001),  while  children  in  the  second  and  third  quartiles  were  significantly 
more  likely  to  experience  adoption  than  those  in  the  first  quartile  (p<.01  and  p<.001, 
respectively).  Family-Based  Reentry,  Congregate  Care  Reunification,  and  Congregate  Care 
Reentry  models  had  limited  patterns  of  significance,  and/or  unexpected  directions  in  the  signs 
of  relationships  emerging  from  adding  Compliance/Timeliness  scores  to  the  basic  models. 

Agency  service  Quality  measures  also  appears  to  have  little  significant  relationship  on 
child  foster  care  outcomes  in  these  models,  and  many  relationships  again  fall  in  unexpected 
directions.  Most  notable  here  is  the  strongly  unexpected  finding  that  children  in  agencies  in 
the  second,  third  and  fourth  quartiles  (reflecting  higher  Quality  scores)  were  much  less  likely 
to  be  adopted  than  those  in  the  lowest  Quality  quartile  (p<.001  in  all  cases).  Also  contrary  to 
expectations,  children  in  second  and  fourth  quartile  of  congregate  care  agencies  had  lower 
reunification  rates  than  those  in  the  lowest  quartile  (p<.01  and  p<.05,  respectively).  As  noted 
above,  this  last  result  does  not  necessarily  have  to  be  interpreted  as  a  negative  outcome,  since 
such  reduced  reunification  may  be  associated  with  more  stable  placements  and  fewer 
absences  from  care  than  may  otherwise  be  typical  in  congregate  settings.  Family-based 
Reunification  and  the  Reentry  models  revealed  few  significant  patterns  of  relationship. 

Introducing  Compliance/Timeliness  and  Quality  measures  simultaneously  into  the 


model  shows  strong  improvements  in  the  likelihood  of  children  reunifying  from  family-based 
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care  as  both  Compliance/Timeliness  and  Quality  scores  rise  (p<.01  for 
Compliance/Timeliness  Quartile,  p<.001  for  all  others).  However,  the  other  four  models 
exhibit  few  statistically  significant  relationships,  and  many  hazard  ratio  scores  falling  in  the 
opposite  of  intended  directions. 

Table  Four  also  presents  data  on  the  number  of  service  agencies  with  significant 
impacts  on  foster  care  outcomes  across  the  different  models.  While  the  number  of  congregate 
care  agencies  with  significant  impacts  on  reunification  and  re-entry  is  little  changed  firom  the 
base  model,  including  Compliance/Timeliness  and  Quality  measures  increases  the  number  of 
family-based  service  agencies  with  significant  positive  and  negative  impacts  on  reunification 
(fi-om  9  to  26),  reentry  (from  2  to  29)  and  adoption  (from  5  to  20).  This  is  again  contrary  to 
expectations — ^well  measured  Compliance/Timeliness  and  Quality  constructs  would  be 
expected  to  absorb  much  of  the  explanatory  power  of  agency  effects,  and  reduce  the  number 
of  agencies  with  significant  outcome  coefficients.  The  instability  in  these  agency  dummy 
variables  would  be  consistent  with  a  model  having  a  specification  error,  such  as  having  an 
important  variable  that  has  systematic  effects  on  outcomes  for  children  in  this  system  being 
unmeasured. 

Results  from  analyses  in  which  two  different  sets  of  sub-scales  within  the  Quality 
measure  are  assessed  for  their  impact  on  child  foster  care  outcomes.  The  first  of  these  uses 
the  three  subscales  used  to  develop  the  overall  Quality  measure — Basic  Welfare,  Normal 
Growth  and  Development;  and  Administrative  Process.  These  subscales  are  described  above 
and  in  Figure  1.  While  many  of  the  results  of  this  analysis  did  reach  statistically  significant 
levels,  few  fell  in  the  expected  direction.  The  only  patterns  in  which  results  were  both 
statistically  significant  and  fell  in  the  expected  direction  were;  higher  Quality/Normal  Growth 

and  Development  scores  where  related  to  improved  Reentry  Outcomes  in  both  Family-Based 
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and  Congregate  Care  models;  and  higher  Quality/Administrative  Process  and  was  associated 
with  improved  Adoption  Outcomes  for  the  highest  quartile  of  care. 

Finally,  Table  Four  presents  results  from  Outcome  models  which  include  four 
alternative  scales  developed  by  ACS  [Edell,  2001].  These  scales  are  derived  from  the  Quality 
of  Care  case  review  tool,  and  tabulate  agency  performance  in  four  specific  domains — 
Assessments,  Service  Planning,  Casework  Contacts,  and  Reunification  Services.  One  notable 
result  is  the  finding  that  agencies  in  the  second,  third  and  highest  quartiles  on  Assessment  had 
better  Reunification  Rates  from  Family  Care  (p<.001).  Even  more  striking  was  the 
relationship  impact  of  Casework  Contacts  on  the  Congregate  Care  models — ^higher  Contact 
scores  were  significantly  associated  with  improved  Reunification  rates  (p<.05,  p<.001,  p<.01 
across  quartiles  2,  3  and  4)  and  reduced  Reentries  (p<.01,  p<.05  and  p<.01  across  quartiles  2, 
3,  and  4).  Again,  most  other  results  of  analyzing  the  alternative  Quality  scales  revealed  few 
other  patterns  of  statistically  significant  results  falling  in  expected  directions.  Particularly 
notable  examples  of  such  unexpected  relationships  include  reunification  services  being 
associated  with  reduced  likelihood  of  reunification  from  family-based  care,  and  service 
planning  being  associated  in  most  cases  with  reduced  reunification  and  increased  re-entry  in 
both  family-based  and  congregate  care  models. 

Impacts  of  model  specification  on  agency  effects:  To  understand  the  impact 
different  specifications  of  these  models  have  on  effect  sizes  and  significance  for  service 
agencies,  results  for  four  randomly  selected  agencies  were  assembled.  Table  Five  presents 
hazard  ratios  and  significance  levels  for  four  agencies,  across  three  model  forms:  “base” 
model  with  no  community  need  scores;  “base”  model  with  community  need  scores;  and 
“base”  model  with  community  need  scores.  Quality  of  Care  and  Timeliness  Measures. 
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This  analysis  demonstrates  the  substantial  shifts  in  the  direction,  magnitude  and 
significance  of  hazard  ratios  for  specific  providers  under  alternative  model  specifications.  For 
example,  hazard  ratio  values  and  patterns  of  significance  for  Agency  1  increase  as  more  data 
are  included  in  the  models.  However,  the  other  three  agencies  see  much  less  consistent 
effects.  Specifically,  each  agency  has  hazard  ratios  that  shift  in  opposite  directions  as  one 
moves  from  the  Base  model  to  the  most  completely  specified  version — ^for  example.  Agency 
2  has  its  hazard  ratio  shift  from  1.21  to  1.18  to  0.41  across  these  models,  indicating  that  the 
effect  of  that  agency  on  child  outcomes  shift  from  promoting  reunification  to  signaling 
somewhat  lower  (and  non-significant)  reunification  effects. 


Table  5.  Impact  of  Alternative  model  specifications  on  four  agencies,  Family-Based  Care 
Reunification  Models 


“Base”  Model,  no 
Community  Needs 

“Base”  Model,  with 
Community  Needs 

“Base”  Model,  with 
Community  Need, 
Quality  of  Care  & 
Timeliness 

Agency 

Hazard 

Significance 

Hazard 

Significance 

Hazard 

Significance 

1 

1.10 

1.15 

** 

1.79 

2 

1.21 

** 

1.18 

0.41 

3 

1.03 

0.89 

0.29 

4 

0.74 

0.64 

1.87 

Conclusions 

This  study  seeks  to  examine  relationships  between  administrative 
Compliance/Timeliness  and  Quality  of  Care  data  and  foster  care  outcomes,  seeking  to 
elucidate  which  measures  might  be  most  useful  to  monitor  and  focus  on  in  efforts  to  improve 
outcomes  for  children  in  out-of-home  placement.  The  rationale  for  undertaking  this  analysis 
is  reflected  in  Courtney’s  [1993]  evaluation  framework,  which  emphasizes  the  role  “Program 
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Process  Characteristics”  such  as  timeliness  and  quality  of  care  should  play  in  foster  care 
outcomes  evaluation.  Overall,  gaps  in  data  available  at  this  time  limit  interpretability  of  the 
findings  from  this  analysis,  it  does  offer  slight  support  for  the  impact  of 
Compliance/Timeliness  measures  on  foster  care  outcomes.  Further,  while  overall  Quality  of 
Care  scores  evidenced  little  relation  to  outcomes,  some  subscales  and  individual  indicators 
within  this  component  may  eventually  demonstrate  substantial  utility  when  additional 
appropriate  data  are  available. 

However,  all  results  presented  here  face  important  limitations  in  their  interpretation. 
The  lack  of  appropriate  risk  adjustments  clearly  introduce  risk  of  specification  error  which 
could  bias  hazard  ratio  estimates  reported  from  these  models.  These  errors  arise  where  factors 
such  as  family  and  child  needs,  risks  and  strengths  which  should  be  expected  to  impact  on 
child  and  family  outcomes  are  left  unmeasured.  At  the  same  time,  the  lack  of  risk  adjustment 
also  increases  risk  of  selection  bias,  as  children  with  different  patterns  of  service  need,  risks 
and  strengths  may  be  non-randomly  distributed  across  service  settings.  Without  appropriate 
measures  of  risk  adjustment,  selection  bias  may  result  in  inaccurate  measures  of  association 
between  control  variables  and  outcomes.  While  including  community  need  scores  in  these 
models  should  somewhat  ameliorate  these  limitations,  policy  researchers  working  in  real- 
world  settings  such  as  child  and  family  service  systems  should  be  cautious  in  interpreting 
results  presented  here. 

A  second  set  of  limitations  on  this  work  lies  in  the  different  sampling  frames  used  in 
collecting  the  data  for  the  Quality  and  Outcomes  datasets.  As  noted  above,  the  Quality  of 
Care  data  were  an  “in-care”  sample,  collected  from  random  samples  of  children  within  each 
program  type  provided  by  each  service  agency,  while  Outcomes  were  calculated  based  on 


cohorts  of  children  entering  or  leaving  care  in  a  given  range  of  years.  While  the  data  used  in 
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these  analyses  were  weighted  to  adjust  for  potential  bias  in  merging  these  datasets,  the  need 
for  such  weighting  would  have  been  reduced  or  eliminated  if  Quality  of  Care  data  were 
collected  from  admission  and  discharge  cohorts  similar  to  or  matching  that  of  the  Outcomes 
analysis. 

The  results  presented  in  this  paper  appear  to  offer  limited  but  positive  support  for  the 
use  of  the  overall  ACS  Compliance/Timeliness  measures,  but  less  support  for  the  Quality  of 
Care  measures  or  its  subscales.  Correlation  analysis  reveals  a  positive  and  significant 
relationship  between  overall  Compliance/Timeliness  scores  and  overall  Outcomes  scores 
(Pearson’s  n=0.59,  p=0.(X),  two  tailed),  and  average  reunification,  re-entry  and  adoption 
finalization  rates  rise  consistently  from  lowest  to  highest  quartile  of  performance.  Agency 
level  proportional  hazards  analysis  reveals  generally  significant  relationships  between 
Compliance/Timeliness  scores  and  most  of  the  Outcome  variables  for  family-based 
reunification  and  adoption  programs.  However,  results  for  the  Congregate  Re-entry  and 
Reunification  analysis  are  inconclusive  or  consistently  fall  in  the  non-expected  direction. 

Much  less  support  is  found  for  the  impact  of  Quality  of  Care  on  foster  care  outcomes. 
While  overall  Quality  scores  are  significantly  correlated  with  overall  Compliance/Timeliness 
scores  (Pearson’s  n=0.31,  p=0.02,  two  tailed),  no  significant  correlations  were  found  between 
Quality  scores  and  overall  Outcomes.  There  are  also  few  consistent  patterns  in  mean  outcome 
performances  across  quartiles  of  Quality  scores.  Most  Agency  level  proportional  hazards 
results  are  non-significant,  and  the  only  strongly  significant  result  (Adoption  Finahzations  at 
the  Agency  Level)  has  hazard  ratio  results  below  1— indicating  that  higher  Quality  scores  are 
associated  with  lower  adoption  finalization  results. 


Evaluating  Foster  Care  Services 


114 


Introducing  both  Quality  and  Compliance/Timeliness  quartile  scores  in  the  Agency 
level  models  does  notably  improve  overall  results  for  two  of  the  outcome  models.  Rising 
Compliance/Timeliness  and  Quality  Scores  are  significantly  associated  with  increases  in 
family-based  reunification  rates,  and  with  decreases  in  family-based  re-entry  rates  when 
entered  simultaneously.  However,  the  other  models  remain  inconsistent  or  non-significant 
when  both  measures  are  introduced.  Finally,  there  is  limited  evidence  that  some  of  the  sub¬ 
scales  within  the  Quality  measure  may  be  associated  with  positive  outcomes  in  some  cases. 
Substantially  significant  relationships  include:  Normal  Growth  and  Development  and 
Reunification  outcomes;  Administrative  Process  on  Adoption;  Assessments  on  Family-Care 
Reunifications,  and  Contacts  on  Congregate  Care  Reunification  and  Reentry  Rates. 

Future  Steps;  Overall,  the  methodology  described  in  this  paper  appears  to  suggest 
some  opportunities  to  strengthen  program  evaluation  of  child  welfare  foster  care  services, 
though  the  specific  approaches  presented  here  each  should  be  subject  to  continuing 
refinement.  The  most  important  area  of  focus  in  this  area  would  appear  to  be  development  of 
risk  adjustments  for  children  and  families  served  by  the  foster  care  systems  under  evaluation. 
Incorporating  data  on  child  and  family  service  needs  from  clinical  and/or  casework 
assessment  tools  into  the  Outcome  models  might  offer  substantial  improvement  in  their 
predictive  power  [Hodges,  1997;  Kirk,  and  Ashcraft,  1998].  Another  approach  to  identifying 
service  needs  could  emerge  from  a  structured  case  review  process,  such  as  the  Matching 
Needs  and  Services  methodology  developed  by  the  Dartington  Social  Research  Group  in 
England  [Dartington,  2000;  Melamid  and  Brodbar,  2001].  If  feasible,  any  methodology  used 
for  risk  adjustment  should  have  high  “test-retest”  validity  or  other  mechanisms  to  permit 
reliable  collection  of  data  over  multiple  time  periods.  Beyond  allowing  for  risk  adjustment  in 
the  existing  Outcome  models,  such  data  would  provide  broader  measures  of  the  Outcomes 
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themselves,  including  the  extent  to  which  child  and  family  risks  and  needs  and/or  well-being 
have  been  shifted  during  the  service  period. 

A  second  future  direction  for  this  work  would  be  to  replicate  these  models  using  data 
on  Quality  of  Care  delivered  to  individual  children.  While  the  sampling  methodologies 
currently  in  place  do  not  permit  development  of  such  models  at  this  time,  individual  level 
analysis  should  be  possible  if  future  cycles  of  Quality  data  collection  could  be  better  aligned 
with  the  admission  and  discharge  cohorts  used  in  the  Outcomes  models.  Such  data  would 
permit  much  more  refined  analysis  of  the  impacts  not  only  of  Quality  subscales,  but  even 
individual  case  review  items  on  child  foster  care  Outcomes.  This  would  also  support  further 
refinement  of  the  Quality  of  Care  tool,  and  could  increase  the  consistency,  reliability,  validity 
and  efficiency  of  this  measure.  At  this  time,  however,  the  results  presented  here  offer  little 
empirical  evidence  to  support  the  Basic  Welfare,  Normal  Growth  and  Process  subscales 
underlying  the  Quality  measure.  If  clear  empirical  evidence  does  not  emerge  from  further 
review  of  these  tools,  this  would  suggest  collapsing  these  scales  into  a  single  qualitative 
program  indicator. 

Finally,  on-going  development  efforts  are  also  required  with  regard  to  the  Quantitative 
Outcomes  data.  This  should  particularly  focus  on  expanding  these  indicators  to  include  other 
measures,  such  as  transfers  or  other  exits  pathways  that  children  take  from  the  foster  care 
system.  Perhaps  even  more  important,  these  measures  should  be  expanded  to  include  more 
complete  populations  of  children,  especially  those  experiencing  multiple  admissions.  Finally, 
efforts  should  continue  to  develop  review  tools  designed  to  focus  on  children  emancipating 
out  of  foster  care  to  independent  living,  who  are  currently  not  covered  these  outcome  models. 
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Chapter  8.  What’s  a  Policy  Researcher  to  do?  Conclusions,  Cross-Cutting  Issues  and 
Future  Research 

The  range  of  issues  presented  in  this  dissertation  is  quite  broad,  and  can  only  represent 
a  small  share  of  the  issues  of  concern  to  those  interested  in  integrated,  community-based 
services  for  at  risk  children  and  their  families.  While  the  conclusions  that  can  be  drawn  from 
this  work  can  only  be  limited  to  the  range  of  issues  presented  here,  some  patterns  which  occur 
across  these  studies  may  provide  support  or  understanding  of  broader  issues  which  impact  on 
services  to  vulnerable  children,  families,  and  communities.  In  addition,  the  work  presented 
here  may  offer  some  insights  of  benefit  to  policy  researchers  active  in  these  types  of  research 
settings. 

This  section  of  this  dissertation  explores  these  issues,  identifying  cross  cutting  themes 
and  issues  of  methodology,  policy,  and  service  practice  that  may  have  broader  implications 
for  the  structure  and  operation  of  these  services.  Finally,  this  section  offers  suggestions  for 
future  research  in  this  area  which  may  be  of  benefit  in  clarifying  issues  and  improving 
operations  of  integrated,  comprehensive  community-based  child  and  family  services,  and 
considers  how  policy  researchers  might  contribute  to  this  work. 

Response  to  Overarching  Research  Questions:  This  work  presents  results  from 
four  separate  analyses,  using  a  wide  variety  of  data  sources  to  expand  the  range  of  program 
evaluation  methods  available  for  child  and  family  services.  While  interpretation  of  results 
from  each  of  these  analyses  is  constrained  by  a  variety  of  data  and  methodological  limitations, 
each  analysis  does  appear  to  offer  some  potential  benefit  to  service  systems.  Viewed  from  the 
perspective  of  the  Courtney  evaluation  framework,  some  of  the  major  findings  across  these 
projects  include: 


Evaluating  Foster  Care  Services 


117 


•  Client  Status  Measures  (Need/Risk  Case  Mix  Adjustments):  Incorporating 
community-level  need/risk  scales  into  event  history  models  of  appears  to  have  some 
utility  in  improving  these  models  sensitivity  to  differential  risks  among  clients 
served  by  these  agencies.  Consensus  based  review  and  development  strategies  may 
provide  improved  ability  to  develop  such  measures.  Peer  based  case  review  tools 
such  as  the  Matching  Needs  and  Services  process  may  support  development  of 
enhanced  measures  of  child  and  family  service  needs;  if  these  such  needs  data  could 
be  collected  and  reported  systematically  for  children  in  care  with  a  given  system, 
these  might  offer  substantial  benefit  as  Client  Status  Measures  in  future  Outcomes 
evaluations.  However,  until  more  comprehensive  assessment  data  on  child-  and 
family-level  needs,  risks  and  strengths  are  available  within  these  models,  these 
findings  must  be  interpreted  with  caution. 

•  Program  Process  Characteristics  (Compliance/TimeUness  and  Quality  of  Care 
Measures):  Limitations  of  missing  data  and  non-overlapping  sampling  frames 
strongly  limit  the  degree  of  confidence  that  can  be  expressed  regarding  results  from 
including  Program  Process  data  from  the  ACS  Compliance/Timeliness  and  Quality 
of  Care  systems  into  the  foster  care  Outcomes  evaluation  models.  Specifically, 
some  evidence  emerges  from  this  work  pointing  towards  associations  between 
program  level  Compliance/Timeliness  scores  and  foster  care  Outcomes.  Little 
association  is  evinced  between  agency-level  Quality  of  Care  scores  and  child 
Outcomes.  However,  this  analysis  reveals  some  suggestion  that  specific  aspects  of 
Quality  of  Care  may  impact  certain  Outcomes — for  example,  results  presented  here 
point  towards  some  positive  associations  between  timely,  comprehensive 
Assessments  and  Reunifications  from  Family  Foster  Care,  and  that  increasing 
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performance  on  Casework  Contacts  in  Congregate  Care  settings  are  associated  with 
increased  Reunifications  and  declining  Re-Entries. 

•  Case  Status  Outcomes  Measures  and  Program  Structural  Characteristics:  The 
analyses  presented  here  build  upon  existing  event  history  Outcomes  evaluation 
models  previously  developed  by  ACS  and  the  Chapin  Hall  Center  for  Children 
[Wulczyn,  Orlebeke  and  Melamid,  2000].  As  such,  these  analyses  have  less  to  offer 
directly  relating  to  these  two  components  of  these  extant  models.  However,  it  is 
worthy  of  note  that  these  base  models  did  remain  relatively  robust  in  the  face  of  the 
additional  data  introduced  into  them  in  these  analyses.  Future  efforts  in  this  area 
should  include  expanding  the  range  of  Outcome  measures  somewhat,  particularly  to 
evaluate  patterns  of  children  transferring  across  multiple  placements  while  in  care, 
as  well  as  unapproved  child  absences  from  care  (AWOLs). 

One  finding  relative  to  Program  Structure  is  also  notable:  when  Outcomes  models 
in  Family-based  foster  care  are  extended  to  include  Community  Needs,  the  number 
of  service  agencies  with  Reunification  and  Re-Entry  rates  above  or  below  average  at 
even  marginally  statistically  significant  levels  declines  somewhat.  If  confirmed  by 
later  studies  having  less  missing  Community  Need  data  and/or  enhanced  child  and 
family  Risk  Adjustment  data,  this  finding  would  support  the  observation  that  child, 
family  and/or  community  needs  might  be  more  important  in  determining  child  foster 
care  outcomes  than  the  characteristics  of  the  provider  agency. 

•  Client  Satisfaction:  While  none  of  the  analyses  presented  here  explicitly  explores 
data  on  Client  Satisfaction,  the  Child-  and  Parent- Validation  Cycles  in  the  Matching 
Needs  and  Services  process  does  appear  to  be  a  somewhat  innovative  approach  to 
expanding  the  role(s)  clients  may  play  in  designing  and  evaluating  these  systems. 
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What  is  a  Policy  Researcher  to  do?  Implications  for  research  in  conununity-hased  chOd 
and  family  service  settings. 

The  range  of  research  projects  presented  in  this  dissertation  reflects  only  a  small  share 
of  the  challenges  confronting  child  and  family  service  systems  undertaking  the  transformation 
to  integrated,  community-based  service  models.  In  addition,  each  of  the  research  protocols 
described  here  remain,  to  different  degrees,  works-in-progress.  Greater  attention  continues  to 
be  required  both  to  the  methodological  and  policy/practice  implications  of  this  work. 
However,  the  work  presented  here  suggests  several  priorities  for  future  research  in  this  area, 
in  supporting  the  creation  of  effective,  comprehensive,  community-based  networks  of  care. 

This  dissertation  may  offer  some  insights  into  the  work  policy  research  and  policy 
researchers  can  play  in  these  settings.  One  clear  observation  across  the  range  of  analyses 
presented  here  is  that  each  of  the  research  and  analysis  initiatives  described  in  this  dissertation 
was  successfully  implemented,  and  were  widely  recognized  as  vital  supports  to  the  overall 
agency  reform  process.  This  work  offers  clear  support  for  the  roles  policy  research  can 
assume  in  social  and  health  service  systems  undertaking  community-based  reforms,  and 
should  offer  encouragement  to  policy  researchers  interested  in  undertaking  rigorous  work  in 
this  field. 

However,  also  suggests  some  of  the  limits  or  bounds  on  policy  research  in  these 
settings.  The  broadest  of  these  boundaries  may  be  the  beliefs  and  attitudes  of  other  actors 
impacted  by  these  systems:  the  range  of  projects  described  herein  was  determined  largely  by 
the  decisions  of  senior  management  involved  in  developing  the  reform  plan  for  this 
jurisdiction,  informed  by  the  input  of  a  wide  variety  of  advocates,  community  members  and 
political  leaders  with  substantial  interests  in  this  effort.  In  a  sense,  then,  policy  researchers  in 
this  setting  undertook  the  work  they  were  asked  to  do,  and  understood  that  this  work 
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constituted  part  of  a  larger  reform  initiative  many  aspects  of  which  they  would  have  only 
indirect  impact  upon. 

The  research  literature  in  this  field  may  offer  some  context  to  understand  the  range  of 
issue  areas  which  policy  researchers  explored  in  this  setting,  and  those  which  received  less 
attention  than  might  be  expected.  For  policy  researchers  currently  engaged  with,  or 
contemplating  beginning  work  in  such  settings,  this  work  may  offer  some  suggestions  for 
directions  to  pursue.  The  most  important  observation  is  that  such  work  is  doable,  and  should 
be  done  in  any  system  serving  children  and  families  with  complex  needs. 

The  need  for  these  evaluation  systems  appears  likely  to  grow  in  the  future,  driven  both 
by  greater  calls  for  accountability  for  the  performance  of  these  systems,  and  by  the  advent  of 
more  distributed,  community-based  service  models.  Under  such  decentralized  systems,  the 
need  for  comprehensive,  comprehensible  data  on  performance  will  grow.  In  fact,  the  ability 
to  safely  track  systems  performance  under  community-based  strategies  may  be  one  major 
constraint  working  against  those  who  seek  to  shift  these  systems  into  such  networks. 

So  what  is  a  policy  researcher  to  do?  Many  of  the  methods  presented  in  this  work 
would  appear  to  have  strongly  generalizable  utility  in  a  wide  range  of  service  settings.  These 
include  development  of  community  risk  scales  to  assess  the  patterns  of  resource  allocation 
across  service  systems,  and  the  development  of  event  history  outcomes  models  to  assess 
performance  across  multiple  programs  and  providers  operating  in  these  settings.  Other 
strategies  employed  in  this  work  may  have  broader  utility  but  must  be  considered  carefully 
prior  to  adoption  in  other  settings.  One  set  of  powerful  strategies  that  must  be  implemented 
with  care  may  be  the  Matching  Needs  and  Services  analysis,  which  reveals  some  lack  of 
consistency  in  the  results  presented  here,  and  has  had  mixed  success  in  leading  to  actual 
changes  in  services. 
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Still  other  strategies  employed  here  appear  to  offer  limited  utility,  and  their  limitations 
should  probably  be  strongly  examined  before  they  are  adopted.  The  most  notable  of  these  is 
are  the  Quality  of  Care  measures,  which  revealed  few  clear  relationships  to  Outcomes  in  this 
analysis,  and  whose  complexity  of  implementation  and  cost  of  operations  are  quite 
substantial.  However,  one  point  of  advice  could  be  offered  to  any  service  agency  that  is  at  an 
earlier  stage  in  developing  evaluation  models — ^individual  level  risk  adjustments  are  required 
in  any  comprehensive  evaluation  tool,  and  should  probably  take  priority  over  development  of 
quality  of  care  measures  to  these  represent  extent  competing  resource  demands. 

The  most  notable  domain  suggested  by  the  earlier  research  which  was  not  explicitly 
addressed  by  the  policy  research  presented  here  falls  within  the  implementation  and  operation 
of  service  programs:  this  research  offers  little  direct  information  to  support  improvements  in 
clinical  assessment,  casework  practice,  staff  development  or  organizational  design.  Instead 
the  research  conducted  in  this  setting  appeared  to  focus  on  the  “front  end”  of  the  reform 
process,  including  analysis  to  support  improved  allocation  of  available  resources  and  service 
design/planning,  and  the  “back  end”  via  effective  program  evaluations. 

Several  avenues  could  be  considered,  if  policy  researchers  were  interested  in  being 
asked  to  participate  more  directly  in  issues  of  program  implementation  and  operation.  One 
approach  may  be  to  explicitly  build  on  the  areas  where  this  and  other  research  has 
demonstrated  an  ability  to  contribute  to  the  field.  For  example,  changes  in  service  process 
such  as  case  practices,  programs,  improvements  in  staff  skill  or  qualifications,  or 
organizational  designs  all  may  be  measurable,  and  their  impact  could  be  examined  through 
process/outcomes  evaluation  methods  similar  to  those  described  in  this  work. 

In  addition,  the  consensus-based  analytic  techniques  underlying  the  resource  allocation  and 
need  group  analyses  described  in  this  dissertation  could  serve  as  starting  points  for  additional 
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research  into  clinical  or  service  need  assessment  technologies,  perhaps  by  systematically 
reviewing  a  sample  of  cases  which  fall  within  different  need  groups  to  more  rigorously 
determine  what  child  and  family  characteristics  or  experiences  can  more  reliably  be  used  to 
define  need  group  membership.  Such  reviews  could  also  offer  insight  into  what  service 
response  is  appropriate  to  each  group  of  clients,  which  could  support  design  of  a  differential 
response  service  system. 

Cross-cutting  Methodological  Issues 

Sampling  Frames/Data  Limitations:  One  set  of  key  issues  arising  from  the  work 
presented  here  are  the  limitations  in  available  data  and  data  collection  methods  used  in  this 
setting.  The  limitations  were  particularly  notable  with  regard  to  final  paper  on  Foster  Care 
Evaluations,  which  sought  to  integrate  Quality  of  Care  and  Administrative 
Compliance/Timeliness  data  into  the  Outcomes  evaluation  models.  The  main  source  of  these 
limitations  lies  in  the  Quality  of  Care  data,  which  used  an  in-care  sampling  frame  rather  than 
relying  on  admission  and/or  discharge  cohorts  that  are  more  reliably  representative  of  the 
child  populations  under  care  in  the  systems  under  study. 

Adopting  a  sampling  frame  more  closely  matching  the  admission  and  discharge  cohort 
sampling  methods  used  in  the  Outcomes  analyses  would  substantially  increase  the  number  of 
cases  with  both  Quality  Of  Care  and  Outcomes  data.  This  would  have  numerous  benefits  on 
these  models,  including  reducing  the  impact  of  weighting  schemes  on  models  specifications, 
permitting  analysis  of  individual  level  Quality  of  Care  on  child  outcomes,  and  increasing  the 
number  of  observations  included  in  each  model.  In  addition,  models  could  be  expanded  to 
include  other  measurable  outcomes  such  as  transfers  to  new  placements,  children  absent 
without  leave  (AWOL),  and  children  who  exit  through  other  pathways  such  as  emancipation 
to  independent  living. 
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Integrating  Qualitative  and  Quantitative  Data:  Despite  these  limitations,  a 
consistent  theme  across  the  four  studies  presented  here  is  the  strong  potential  available  from 
integrating  data  from  multiple  sources,  and  in  using  multiple  research  methodologies  in  order 
to  provide  a  more  complete  portrait  of  the  service  needs  and  outcomes  of  multi-challenged 
children  and  families.  For  example,  while  the  paper  on  Enhancing  Equity  in  Community 
Based  Services  uses  standard  quantitative  methodologies  to  develop  a  multidimensional  scale 
representing  community  need  across  59  New  York  City  neighborhoods,  scale  weights  were 
developed  using  qualitative  consensus-based  decision-making  processes.  Similarly,  while  the 
work  on  Matching  Needs  and  Services  used  team-based  qualitative  case  record  reviews  to 
identify  patterns  of  client  service  need,  the  success  of  this  methodology  was  based  on 
carefully  considered  and  properly  executed  case  sampling  designs  to  ensure  appropriate 
randomization  and  representative  sampling  of  cases  within  targeted  communities. 

However,  the  clearest  example  of  the  benefit  of  integrating  qualitative  and  quantitative 
methods  presented  in  this  body  of  work  is  probably  the  final  paper  on  Evaluating  Foster  Care 
Outcomes.  This  work  sought  to  incorporate  results  from  quaUtative  Quality  of  Care  and 
Administrative  Compliance/Timeliness  measures  into  an  existing  quantitative  event  history 
Outcomes  models  in  an  attempt  to  strengthen  the  predictive  quality  of  these  models. 
Unfortunately,  the  results  of  this  analysis  demonstrated  little  consistent  benefit — while  some 
confirmation  was  found  regarding  the  impact  of  Administrative  Compliance/Timeliness  on 
child  foster  care  Outcomes,  the  Quality  of  Care  measures  revealed  no  clear  relationships. 

This  reflects  the  continued  challenge  of  integrating  qualitative  and  quantitative  methods  in  the 
real-world  context  of  services  to  children  and  families,  and  some  of  the  potential  hmitations 
of  this  apparently  powerful  combination  of  approaches. 


Evaluating  Foster  Care  Services 


124 


Incorporating  Risk  Adjustment:  Perhaps  the  single  most  important  methodological 
contribution  of  this  work  emerges  from  the  integration  of  community  needs  scores  into  the 
evaluation  models  as  reported  in  the  paper  on  Effects  of  Community  Level  Needs  on  Foster 
Care  Outcomes.  In  this  work  community  need  scores  calculated  using  the  methodology 
described  in  the  first  analytic  chapter  of  this  work  are  integrated  in  the  existing  Outcomes 
models,  using  event  history  analysis.  This  work  addresses  a  commonly  recognized  limitation 
of  service  evaluation  models:  the  lack  of  adequate  risk  adjustment  to  appropriately  reflect  the 
needs,  challenges,  and  unique  circumstances  of  individual  children  and  families  served  by 
these  programs  [Courtney,  1993].  In  this  case  community  need  scores  are  introduced  has 
proxy  measures  of  these  child  and  family  risks,  since  more  appropriate  measures  of  child  and 
family  need  or  well-being  are  not  reliably  or  consistently  reported  in  data  available  in  the 
subject  jurisdiction  [Altshuler  and  Gleeson,  1999]. 

Including  community  needs  scores  in  the  Outcome  models,  while  requiring  propensity 
weights  to  accommodate  missing  data  and  being  only  marginally  statistically  significant  in 
one  of  four  models,  otherwise  exhibits  effects  on  the  models  as  would  be  expected-- 
controlling  for  child  demographic  and  placement  experience  characteristics,  increasing 
community  need  values  are  associated  with  reduced  likelihood  of  reunification  from  both 
family  and  congregate  care  settings.  However,  effects  on  re-entry  models  do  not  follow  the 
expected  direction  and  are  not  statistically  significant.  Introduction  of  community  needs 
scores  into  these  models  does  not  appear  to  create  meaningful  changes  in  patterns  of 
significance  and  directions  of  effect  of  other  control  variables  included  in  these  models. 

The  most  striking  implication  of  including  community  needs  in  these  models  is  their 
effect  on  the  patterns  of  significance  of  the  dummy  variables  for  agency  effects  on  child 
outcomes  in  foster  care:  the  number  of  agencies  with  hazard  ratios  that  are  statistically 
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significant  drops  substantially  for  family  based  agencies  in  both  the  reunification  and  re-entry 
models.  Effects  on  congregate  care  agency  patterns  of  significance  are  minimal.  While  these 
findings  are  strongly  limited  by  their  marginal  statistical  significance,  and  by  the  need  to  use 
propensity  weights  to  adjust  for  patterns  of  missing  community  residency  data  among 
children  in  the  sample  this  dramatic  effect  on  agency  patterns  of  significance  is  difficult  to 
fully  discount.  At  a  minimum,  these  findings  do  appear  to  underscore  the  potentially  strong 
effects  that  well  measured  risk  adjustments  might  have  on  foster  care  outcome  models  if  they 
can  be  appropriately  measured. 

Client  Validation/Customer  Satisfaction:  A  final  methodological  point  worthy  of 
note  emerges  from  the  Matching  Needs  and  Services  process,  which  included  explicit 
attention  to  client  validation  as  part  of  the  analytic  process.  Though  many  reviewers  call  for 
the  inclusion  of  client  satisfaction  measures  in  any  comprehensive  program  evaluation  for 
systems  need  analysis,  the  difficulty  of  accurately  measuring  and  reporting  client  perspectives 
on  services  has  been  widely  recognized  [Wells,  1999;  Courtney,  1993].  While  the  client 
validation  process  used  in  the  Matching  Needs  and  Services  study  does  not  seek  to  assess  the 
level  or  nature  of  client  satisfaction,  it  does  provide  a  highly  efficient  and  apparently  effective 
method  of  providing  client  input,  perspective  and  feedback  into  a  well  structured  systems 
level  service  need  assessment  and  planning  process,  and  may  be  adaptable  in  other  analytic 
contexts. 

Policy  Level  and  Service  Practice  Issues 

Transparent,  Rigorous  Methodologies:  Also  emerging  across  the  studies  reported 
here  are  a  number  of  specific  policy  level  and  service  practice  issues,  as  well  as  indirect 
support  for  some  very  broad  policy  and  practice  initiatives  which  have  gained  substantial 
support  in  recent  years.  The  first  set  of  such  observations  relates  to  the  connection  between 
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methodology  and  policy  level  issues.  The  studies  reported  here  all  reflect  work  undertaken  as 
part  of  a  major  reform  initiative  in  a  single  large  child  welfare  service  jurisdiction.  As  such, 
these  projects  required  substantial  attention  to  the  political  context  and  operational  realities  of 
a  complex,  dynamic  organization,  and  had  to  respond  to  the  demands  of  diverse 
constituencies  including  children  and  families  served  by  this  agency,  agency  managers  and 
staff,  non-profit  service  providers,  community  members,  political  leaders,  and  researchers  and 
evaluators. 

These  products  exhibited  several  characteristics  that  appear  to  contribute  to  their 
success  in  this  complex  environment.  Each  of  these  analyses  sought  to  use  rigorous 
methodologies  appropriate  to  the  analytic  task  at  hand,  which  were  widely  communicated  to 
the  various  audiences  affected.  Results  were  disseminated  using  a  variety  of  formats 
including  pubhshed  reports,  public  presentations,  interviews  with  journalists,  and  publication 
in  peer  reviewed  journals.  Necessary  resources  were  obtained  through  foundation  grants  and 
budgetary  allocations,  providing  appropriate  access  to  technical  assistance  to  ensure  the 
reliability  and  consistency  of  data  analysis.  Specific  attention  was  placed  on  obtaining 
support  and  validation  for  these  methods  from  senior  managers,  and  in  the  case  of  the 
Matching  Needs  and  Services  analysis  children,  parents,  and  line  staff  were  explicitly 
included  as  active  participants  in  the  analytic  process. 

Implementation  of  Results/Resource  Allocations:  Each  of  the  analyses  described  in 
this  dissertation  was  supported  by  the  use  of  rigorous  methodologies  appropriate  to  the  unique 
demands  of  the  programs  and  jurisdiction  undertaking  this  work.  Furthermore  in  paying 
careful  attention  to  clearly  and  consistently  communicating  with  all  affected  or  interested 
policy  actors,  the  policy  researchers  undertaking  this  work  were  able  to  garner  sufficient 
support  for  and  minimize  resistance  to  successful  implementation  of  both  these  specific 
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analytic  strategies  and  the  larger  reform  agenda.  Examples  of  rigorous  methodology 
supporting  effective  implementation  include  the  use  of  administrative  data,  case  record 
reviews,  and  event  history  analysis  to  provide  triangulation  in  evaluating  foster  care  services, 
and  the  use  of  multiple  indicators  assembled  into  well  conceived  scales  to  support  resource 
allocations  across  communities  within  this  jurisdiction.  In  each  case,  research  and  evaluation 
initiatives  expected  to  have  substantial  impacts  on  providers,  conununities  and  clients  were 
widely  supported  by  affected  parties. 

Another  example  of  the  challenge  of  balancing  analytic  methods  with  program 
demands  was  seen  in  the  implementation  of  results  from  the  Matching  Needs  and  Services 
analysis.  In  this  case,  while  the  analysis  indicated  a  wide  range  of  needs  among  clients,  the 
jurisdiction  chose  to  focus  on  housing  needs  which  had  been  revealed  as  a  need  in  every 
review  cycle  and  which  was  likely  to  be  too  complex  and  costly  for  most  community-level 
organizations  to  address  without  support  from  central  management  of  the  jurisdiction.  Again, 
broad  support  for  the  underlying  analytic  processes  allowed  the  jurisdiction  to  address 
challenging  policy  and  programmatic  issues  which  had  previously  received  less  focus. 

This  example  and  the  agency's  success  in  implementing  revised  resource  allocations 
resulting  from  the  Equity  analysis  also  underscores  a  less  generalizable,  but  non-surprising 
observation:  increasing  the  size  of  resources  available  to  support  services  can  help  overcome 
resistance  to  implementation  of  new  program  initiatives.  However,  efforts  to  shift  resources 
to  programs  and/or  communities  with  greater  needs  are  probably  more  valuable  and  more 
effective  during  periods  of  resource  limitation. 

Association  between  Conununity  Needs  and  Reunification  from  Family  Foster 
Care:  As  noted  above,  the  evaluation  models  included  in  this  dissertation  are  strongly  limited 
by  their  failure  to  include  appropriate  risk  adjustments  to  reflect  many  of  the  specific 
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challenges  and  strengths  of  children  and  families  served  in  these  settings.  In  the  case  of  the 
analysis  of  impacts  on  community  needs  on  foster  care  outcomes,  significant  levels  of  missing 
data  on  child  communities  compound  these  model  specification  errors.  While  an  appropriate 
weighting  scheme  has  been  implemented  in  this  study  to  adjust  model  results  for  these 
missing  values,  all  results  from  these  models  must  be  considered  with  caution  at  this  time. 

Nonetheless,  the  finding  of  significant  associations  between  levels  of  community  need 
and  reunification  from  family  foster  care  is  worthy  of  some  note,  as  it  does  provide  empirical 
confirmation  of  a  central  tenet  of  researchers,  systems  managers,  political  and  community 
leaders  and  advocates  who  have  made  calls  for  shifting  service  to  the  community  level  and/or 
targeting  resources  to  communities  based  on  the  level  and  nature  of  their  needs.  However, 
until  evaluation  models  incorporate  additional  data  regarding  individual  level  risks,  needs  and 
strengths  of  families  and  children  served  by  these  systems,  these  results  cannot  be  confirmed 
and  the  precise  mechanisms  through  which  this  relationship  may  be  operating  will  remain 
unclear. 

Effects  of  Service  Agency  on  Foster  Care  Outcomes:  Though  the  above  noted 
limitations  on  available  data  constrain  our  ability  to  fully  understand  its  relationship  to  foster 
care  outcomes,  the  inclusion  of  community  need  in  these  models  does  appear  to  have 
significant  benefit — ^these  measures  serve  conceptually  as  proxy  measures  for  a  range  of 
service  needs  confronting  families  and  children  served  in  these  settings,  while  having  only 
minimal  impact  on  other  variables  already  included  in  these  models.  As  a  result,  errors  in 
hazard  ratios  for  other  variables  included  in  these  models  should  be  reduced,  allowing  the 
models  to  more  appropriately  reflect  the  contribution  each  control  variable  has  on  foster  care 
outcomes. 
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These  enhanced  models  reveal  notable  shifts  in  patterns  of  significance  among  dummy 
variables  for  foster  care  service  agencies,  with  reductions  in  the  number  of  agencies 
exhibiting  significant  effects  on  foster  care  outcomes  from  the  base  models  that  do  not  include 
community  need.  If  validated,  this  result  might  be  expected  to  have  some  impact  on  how 
these  models  have  been  used  in  this  jurisdiction.  In  particular,  ACS  has  used  the  results  of  the 
evaluation  methodologies  described  in  this  dissertation  in  recent  years  to  determine  maximum 
capacity  levels  for  service  agencies,  and  the  agency  director  has  committed  to  terminating 
contracts  with  agencies  consistently  seen  as  poor  performers  [Bernstein,  2001].  In  addition, 
these  models  underlay  the  recently  initiated  Safe  and  Timely  Adoption  and  Reunification 
(STAR)  program  within  this  agency,  which  provides  additional  funding  for  agencies  whose 
reunification,  re-entry  and  adoption  outcomes  exceed  levels  predicted  by  these  models. 

In  combination,  these  uses  of  the  evaluation  models  described  here  can  and  do  have 
dramatic  impacts  on  the  agencies  providing  service  to  approximately  30,000  children  in  care 
in  New  York  City.  This  jurisdiction  or  others  contemplating  implementing  similar  programs 
should  attend  carefully  to  the  potential  limitations  of  these  tools  as  they  stand,  and  take  steps 
to  improve  their  accuracy  wherever  possible.  The  analysis  presented  here  appears  to  suggest 
two  opportunities  for  such  enhancement:  improve  collection  of  home  address  data  for 
children  in  care  to  reduce  patterns  of  missing  community  need  data  in  these  models,  and  make 
every  effort  possible  to  gather  and  include  additional  data  on  child  and  family  service  needs, 
risks  and  strengths  in  these  models. 

Factors  Associated  with  Improved  Service  Outcomes:  Despite  the  many 
limitations  of  the  evaluation  models  underlying  the  two  papers  on  foster  care  outcomes,  this 
work  does  offer  some  suggestions  as  to  factors  associated  with  improved  service  outcomes  for 
children  in  foster  care. 
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Perhaps  the  most  striking  of  these  factors  is  the  relationship  between  community-level 
needs  and  child  outcomes  in  foster  care.  While  multivariate  models  revealed  only  marginally 
significant  relationships  between  community  need  and  reunification  from  family  care,  and 
non-significant  relationships  in  the  other  models,  the  direction  of  relationships  in  the 
multivariate  and  univariate  analyses  did  indicate  that  increasing  community  need  was 
associated  with  reduced  likelihood  of  reunification.  Perhaps  more  importantly,  including 
community  needs  in  these  models  had  the  effect  of  substantially  reducing  the  number  of 
service  provider  in  agencies  with  significant  impacts  on  child  reunification  outcomes.  This 
last  observation  would  appear  to  support  the  perspective  that  child  foster  care  outcomes  might 
be  strongly  affected  by  child,  family  and  community  needs,  and  less  so  by  characteristics  of 
provider  agencies. 

Integrating  service  agency  Quality  of  Care  and  Administrative  Compliance/Timeliness 
data  into  the  Outcomes  models  revealed  few  patterns  of  relationship  between  these  measures 
and  child  reunification,  re-entry,  and  adoption  finalization  rates  in  foster  care.  More 
specifically,  univariate  and  multivariate  analyses  provided  limited  support  for  the  relationship 
between  Administrative  Compliance/Timeliness  measures  and  reunification,  re-entry,  and 
adoption  finalization  outcomes.  Given  the  Administrative  Compliance/Timeliness  measures 
focus  on  timeliness  and  completeness  of  case  work  activities,  this  observation  would  appear 
to  provide  some  validation  for  reliance  on  relatively  easy  to  manage  and  inexpensive 
reporting  systems  as  useful  proxies  for  measures  of  service  quality. 

Unfortunately,  the  more  comprehensive  Quality  of  Care  measures  revealed  no  patterns 
of  relationship  in  univariate  analyses  of  child  outcomes,  and  introducing  these  measures  into 
the  multivariate  Outcomes  models  not  only  revealed  no  significant  relationships  between 
increasing  service  quality  and  child  outcomes,  but  showed  many  relationships  which  fell  in 
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the  opposite  of  expected  directions.  Integrating  both  Quality  and  Compliance/Timeliness 
measures  simultaneously  into  the  multivariate  models  also  revealed  no  significant  or 
consistent  relationship  between  service  quality  and  outcomes,  though  process  measures  did 
appear  to  have  somewhat  stronger  relationships. 

Finally,  one  benefit  of  the  Quality  of  Care  measure  is  the  availability  of  sub-scale 
scores  which  might  permit  greater  decomposition  of  the  factors  underlying  positive  or 
negative  outcomes,  if  they  can  be  found  to  be  systematically  related  to  these  outcomes. 
Analysis  of  the  three  sub-scales  used  to  develop  the  Quality  of  Care  measures  (basic  welfare, 
normal  growth  and  development,  and  service  process),  revealed  only  limited  significant 
relationships  in  the  expected  directions — Quality/Normal  Growth  scores  appeared  to  be 
related  to  improved  reunification  outcomes,  and  Quality/Administrative  Process  was  related 
to  improved  adoption  finalizations. 

However,  these  sub-scales  have  proven  difficult  for  service  agencies  to  understand  and 
use  in  improving  services  to  children  and  families  [Edell,  2001].  This  has  led  analysts 
working  within  this  service  system  to  develop  alternative  sub-scales  using  elements  from  the 
Quality  of  Care  case  record  review  instruments.  Entering  these  alternative  sub-scale  scores 
into  the  multivariate  Outcomes  models  also  reveals  relatively  few  patterns  of  significance, 
though  there  are  some  relationships  of  substantial  interest.  These  include  apparently 
significant  relationships  between  psychosocial  assessments  and  reunifications  from  family 
care,  and  casework  contacts  increasing  reunification  and  decreasing  re-entry  rates  in 
congregate  care.  Unfortunately,  the  lack  of  a  sufficient  number  of  cases  matching  at  the 
individual  child  level  prevents  more  specific  the  composition  of  the  relationships  between 
sub-scales  items  and  child  outcomes  in  foster  care. 
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Risk  Adjustment:  As  noted  in  the  above  discussion  of  cross-cutting  methodological 
issues,  a  key  challenge  in  developing  systems  level  need  assessments  and  program  evaluation 
methodologies  is  the  development  of  appropriate  risk  adjustments  to  ensure  that  models 
control  for  different  risks  facing  individual  children  and  families.  These  risks  arise  from 
patterns  of  need,  child  and  family  strengths  and  resources,  and/or  the  health  and  well-being  of 
children  and  families  served  by  these  systems.  In  incorporating  community  needs  into  the 
existing  Outcomes  models,  the  analyses  presented  here  appear  offers  some  support  for  the 
benefit  of  risk  adjustment  in  evaluations  of  foster  care  outcomes.  However,  such  community 
need  scores  can  only  serve  as  a  proxy  for  broader  measures  of  clinical  need,  child  and  family 
well  being  or  risk,  and  additional  work  is  clearly  required  to  improve  risk  adjustment  in  these 
models. 

Unfortunately,  there  is  little  consensus  among  researchers  and  evaluators  as  to 
appropriate  methodologies  that  can  consistently  and  reliably  be  used  in  child  and  family 
service  settings  to  assess  clinical,  health,  social,  educational,  family  functioning,  and  other 
needs  and  strengths  of  those  served  by  these  programs.  While  there  have  been  a  number  of 
notable  attempts  to  create  clinical  assessment  or  risk  profiling  tools  that  can  better  identify 
patterns  of  risk  for  children  and  families  touched  by  child  welfare  service  systems,  their  use 
remains  inconsistent  and  have  seen  httle  reported  use  as  part  of  large  scale  program 
evaluation  or  service  planning  tools  in  child  and  family  service  settings.  However,  well 
designed,  thorough,  valid  and  consistent  clinical  assessment  technologies  would  likely 
represent  the  “gold  standard”  of  any  risk  adjustment  methodology.  Any  such  methods  having 
strong  “test-retesf  ’  validity  could  also  allow  for  reliable  and  consistent  assessments  of 
changes  in  well-being  and/or  needs  of  children  and  families  served  by  these  systems. 
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The  Matching  Needs  and  Services  analysis  presented  in  this  dissertation  may  provide 
some  insight  into  how  more  reliable,  consistent,  and  usable  assessment  tools  might  be 
developed  and  structured.  By  identifying  patterns  of  need  across  a  representative  sample  of 
children  and  families  and  using  language  that  reflects  the  beliefs  and  understandings  of  staff 
and  clients  served  by  these  systems,  this  method  appears  to  offer  a  potentially  powerful  tool 
for  understanding  the  challenges  facing  children  and  families  at  risk.  While  need  groups 
identified  through  the  Matching  Needs  and  Services  process  may  not  adequately  capture 
complex  health  or  psychological  service  needs  of  at-risk  children  and  families,  they  might 
offer  useable  and  efficient  means  of  having  caseworkers  identify  patterns  of  need  for  children 
and  families  they  serve,  and  for  developing  improved  risk  adjustment  protocols  for  program 
evaluations. 

Patterns  of  Need  among  ChUdren  and  Families:  The  analyses  presented  in  this 
dissertation  also  offer  a  range  of  approaches  to  measuring  and  describing  child-,  family-  and 
community-level  needs  of  clients  who  come  into  contact  with  these  service  systems.  As  noted 
above,  integrating  community  need  scores  into  foster  care  Outcomes  models  revealed  a 
marginally  significant  inverse  relationship  between  community  needs  and  reunification  rates 
in  family  foster  care,  though  other  relationships  were  not  significant.  While  this  relationship 
must  be  considered  a  preliminary  finding  given  the  substantial  levels  of  missing  data  and  the 
lack  of  additional  child-  and  family-level  risk  adjustments  in  these  models,  it  does  appear 
supportive  of  the  role  community-level  need  may  play  in  foster  care  outcomes. 

Much  more  persuasive  measures  of  patterns  of  need  emerge  from  the  Matching  Needs 
and  Services  analysis.  This  analysis  highlighted  a  number  of  specific  patterns  of  presenting 
problems  among  children  and  families  served  in  this  setting.  These  included  needs  in  such 
areas  as  substance  abuse,  adoption  services,  parent  skills  training/supports. 
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emotional/psychological  well-being,  and  housing  services.  More  importantly,  this  analysis 
offers  the  accessible  definitions  of  the  multiple  challenges  and  strengths  of  children  and 
families  who  fall  within  each  of  these  need  groups,  clear  statements  of  desired  outcomes  for 
each  need  group,  and  suggestions  for  service  strategies  and  practices  which  may  be  effective 
in  serving  the  sets  of  families.  In  addition,  the  Matching  Needs  and  Services  process  itself 
appears  to  have  benefit  as  a  tool  to  support  group  decision  making,  especially  in  the  context  of 
implementing  community-based  service  structures  to  address  the  multi-dimensional  and 
complex  needs  of  at-risk  children  and  their  families. 

Integrating  Services  at  the  Community  Level:  Perhaps  the  broadest  implication  of 
the  work  presented  here  may  be  its  indirect  support  for  integrated,  community-based  service 
models  which  might  offer  differential  response  to  children,  families  and  communities  with 
different  patterns  of  service  need.  Such  “differential  response”  structures  would  emphasize 
the  adoption  of  enhanced  client  and  family  need  assessment  technologies  and  the 
development  of  comprehensive,  integrated  networks  of  care  to  ensure  thorough,  timely, 
effective  and  accessible  services  to  vulnerable  children  and  their  families.  Ultimately  such  a 
service  strategy  seeks  to  recognize  that  children  live  in  families  and  that  families  live  in 
communities,  and  that  addressing  complex,  dynamic  social  problems  requires  multifaceted 
service  strategies. 

The  strategies  described  in  this  dissertation  offer  only  a  portion  of  the  full  range  of 
approaches  required  to  support  the  creation  of  integrated,  community-based  social  services 
that  can  provide  differential  response  to  families  and  children  facing  different  patterns  of  risk 
and  need.  However,  each  of  the  analyses  presented  here  does  appear  to  provide  useful 
strategies  to  support  implementation  of  integrated,  community-based  service  networks,  and 
offers  at  least  indirect  support  for  such  strategies.  For  example,  the  work  on  Resource 
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Allocations  in  community-based  service  models  demonstrates  that  even  challenging  resource 
allocation  decisions  can  be  undertaken  successfully,  and  suggests  that  such  improved  equity 
can  be  a  valuable  strategy  to  improve  the  overall  effectiveness  of  community  based  service 
systems. 

Similarly,  the  papers  on  Evaluation  of  Service  Outcomes  in  community-based  service 
models,  while  focused  on  traditional  foster  care  service  programs,  offers  a  range  of  strategies 
which  can  be  used  in  a  variety  of  service  settings  to  determine  effectiveness  of  programs  in 
meeting  the  complex  needs  of  at  risk  children  and  their  families.  These  include:  (a)  the  use 
of  administrative  data  to  support  event  history  analysis  in  multivariate  models  to  assess  child 
and  family  service  Outcomes  while  controlling  for  demographic  and  service  history 
experiences;  (b)  incorporating  measures  of  community  need  as  proxies  to  account  for  the 
different  risks  and  needs  of  children  and  families;  and  (c)  some  limited  support  for  the  use  of 
Administrative  Compliance/Timeliness  data  either  incorporated  directly  in  these  models  or 
used  as  additional  separate  measures  of  service  quality.  Unfortunately,  this  analysis 
demonstrated  little  conclusive  evidence  that  the  Quality  of  Care  measures  used  in  this  setting 
were  effective  in  delineating  meaningful  differences  in  program  quality,  as  measured  by 
reunification,  re-entry  and  adoption  outcomes.  However,  there  was  some  limited  support  for 
the  effectiveness  of  certain  sub-scales  within  the  Quality  of  Care  measures  used  here; 
additional  analysis  of  these  sub-scales  at  the  individual  child  level  might  find  support  for 
these  relationships,  if  a  sufficiently  large,  representative  sample  could  be  matched  across 
these  two  data  collection  systems. 

The  strongest  support  and  clarification  of  the  role  integrated,  community-based 
services  might  play  in  supporting  multi-challenged  families  and  communities  presented  in  this 
dissertation  clearly  emerges  from  the  work  on  Matching  Needs  and  Services.  This  analysis 
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demonstrates  that  while  vulnerable  children  and  their  families  have  a  great  many  interlocking 
needs,  clear  patterns  can  be  identified  and  staff  and  clients  can  reach  consensus  as  to  how  they 
can  best  be  addressed.  This  result  could  underpin  the  development  of  systems  of  care  that  can 
respond  to  the  individual  patterns  of  need  of  groups  of  families,  and  perhaps  serve  as  the  first 
step  towards  a  true  "differential  response"  system  which  would  provide  more  timely,  more 
appropriate  and  more  effective  services  to  these  hard-to-serve  populations. 

Also  notable  in  these  results  is  the  priority  they  place  in  most  cases  on  intensive, 
multiple  services  and  supports  for  the  child's  entire  family— services  which  must  be  accessible 
and  comprehensible  to  these  families.  While  the  Matching  Needs  and  Services  analysis  does 
not  explicitly  provide  a  supporting  rationale  for  the  delivery  of  such  services  in  communities 
where  at-risk  children  and  family  reside,  this  perspective  appears  to  infuse  the  language  and 
recommendations  emerging  from  this  work.  This  latter  observation  is  reflected  in  the  subject 
the  agency's  decision  to  use  this  methodology  in  its  efforts  to  create  integrated,  community- 
based  networks  of  care  in  this  jurisdiction. 

Suggestions  for  Future  Research 

One  area  requiring  substantial  additional  research  clearly  lies  in  evaluation  of  complex 
social  service  programs,  including  foster  care  and  preventive  social  services.  While  the  value 
of  event  history  analysis  as  a  tool  for  quantifying  outcomes  in  multivariate  models  has  been 
increasingly  well  documented,  the  analyses  presented  here  and  the  work  of  other  researchers 
in  this  field  makes  it  clear  there  remains  significant  need  to  broaden  the  range  of  outcome 
measures  included  in  these  models,  and  to  enhance  risk  adjustment  methods  which  can  better 
differentiate  the  level  of  risk  and  types  of  need  facing  children  and  families  under  care.  While 
risk  adjustments  must  be  clinically  valid  and  reliable  if  they  are  to  be  effective,  they  must  also 
be  sufficiently  robust,  efficient,  usable  and  comprehensible  to  ensure  staff  working  in  the 
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systems  can  effectively  use  them.  This  remains  a  substantial  challenge,  which  will  require 
resources  and  attention  if  it  is  to  be  addressed. 

Closely  related  to  the  issue  of  risk  adjustment  is  the  development  of  assessment 
technologies  which  might  support  improved  measurement  and  diagnosis  of  presenting  issues 
and  strengths  of  families  under  care.  While  many  systems  have  implemented  a  variety  of 
child  safety  assessment  protocols  designed  to  support  investigation  of  child  abuse/neglect 
complaints,  somewhat  less  attention  has  been  paid  to  the  development  of  consistent,  valid  and 
reliable  clinical  or  diagnostic  assessment  tools  which  can  be  used  by  casework  staff  providing 
on  going  services  to  children  and  families.  There  remains  substantial  need  in  the  field  for  the 
continued  enhancement  of  available  tools  and  the  development  of  new  tools,  particularly 
those  with  significant  levels  of  "test-retest”  validity  which  could  be  used  to  assess  changes  in 
child  and  family  service  needs  during  the  course  of  care.  Such  tools,  in  addition  to  providing 
improved  risk  adjustment  for  evaluation  purposes,  could  provide  broader  measures  of  client 
outcomes  including  changes  in  level  and  nature  of  need  during  the  course  of  service.  Once 
aggregated,  these  measures  of  service  need  change  could  dramatically  deepen  understanding 
of  what  constitutes  effective  service,  and  which  strategies  are  best  able  to  address  the  needs  of 
different  client  populations. 

Improving  the  ability  to  appropriately  assess  patterns  of  need  across  children  and 
families  served  by  these  systems  will  also  support  development  of  "differential  response" 
systems  that  would  better  target  services  to  clients  with  different  needs.  The  work  presented 
here  from  the  Matching  Needs  and  Services  analysis  provides  one  portrait  of  what  these  need 
patterns  appear  to  look  like,  reveahng  that  families  served  by  these  systems  have  diverse 
challenges  and  strengths  which  may  require  different  service  priorities.  In  particular,  this 
work  appears  to  suggest  some  specific  areas  where  further  exploration,  development  and 
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adoption  of  alternative  service  strategies  would  be  well  worthy  of  further  consideration. 
These  include  services  to  families  facing  challenges  in  areas  such  as  substance  abuse, 
emotional  health,  parenting  skills,  domestic  violence  and  housing. 

Ultimately,  the  value  of  integrated,  community-based  networks  of  care  may  not  be 
assessable  until  a  wide  range  of  supports  and  analytic  tools  are  put  in  place  and 
comprehensively  examined  in  a  number  of  jurisdictions.  Specific  components  which  will  be 
required  in  this  broad  evaluation  include:  appropriate  and  timely  evaluation  structures; 
reliable  and  consistent  risk  adjustment  systems,  particularly  those  based  on  comprehensive, 
clinically  valid  child  and  family  assessment  technologies;  service  practices  which  emphasize 
differential  response  based  upon  these  assessments;  and  the  implementation  of 
comprehensive,  integrated  networks  of  care  which  can  provide  the  full  range  of  services  to 
families  and  children  in  the  communities  where  they  live.  Until  such  work  is  completed  it 
remains  vital  that  policy  actors  operating  in  these  settings  continue  to  demand 
methodologically  rigorous  and  responsive  research  in  this  arena. 
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